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ABSTRACT; 


Project EOS is studying the problems of building adaptable real-time 
embedded operating systems for the scientific missions of NASA. Choices, a Class 
Hierarchical Open Interface for Custom Embedded Systems, is an operating sys- 
tem designed and built by Project EOS to address the following specific issues: 
the software architecture for adaptable embedded parallel operating sys- 
tems, the achievement of high-performance and real-time operation, the 
simplification of interprocess communications, the isolation of operating 
system mechanisms from one another, and the separation of mechanisms 
from policy decisions. Choices is written in C++ and runs on a ten processor 
Encore Multimax. The system is intended for use in constructing specialized 
computer applications and research on advanced operating system features 
including fault-tolerance and parallelism. 

One of the applications made possible by our research is a software system 
that allows workstation applications to be closely integrated with software run- 
ning on specialized computers like a supercomputer or supermini. CLASP is a 
mechanism that allows the virtual memory space of a workstation to be 
shared with a high-performance computer. CLASP implements a cross- 
architecture procedure call that allows an application on a workstation tran- 
sparently to invoke procedures on the high-performance machine. The method 
allows existing software packages to be decomposed without change onto a 
compatible workstation supercomputer or supermini computer pair. Ray Essick’s 
Ph.D. thesis documents this work. 
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1. Introduction. 

Project EOS is investigating the design and construction of embedded real-time 
systems for applications in NASA’s aerospace programs. The results of our study in pre- 
vious years is documented in the bibliography in Appendix A. In the first six months of 
the current grant period, we built a prototype adaptable embedded real-time operating 
system for parallel computers called Choices, designed and built CLASP, a mechanism 
that uses virtual memory to implement a flexible remote procedure call, and, to satisfy 
several requests for previous Project EOS work, created a new release of the Path Pascal 
compiler for Berkeley UNIX® BSD 4.3. An interface compiler for Choices has been 
designed, but is not yet implemented. 


2. Choices 

Choices is an experimental real-time embedded operating system for parallel and 
distributed computer systems in aerospace applications. The initial prototype has been 
built on a ten processor Encore Multimax. The system is designed to support: 

• the object-oriented organization of user applications, 

• applications requiring custom designed operating systems, 

• diverse hardware architectures (both networked computers and shared memory 
multiprocessors), 

• parallel computation where performance is an issue, 

• persistent objects, 

• protection, 

• real-time operation of applications, 

• research and applications requiring specialized operating system functions. 

The design of the operating system reflects an object-oriented approach. The code 
is organized to meet a number of objectives: 

• The software is to be placed in the public domain. 

• The software is organized as a hierarchy of classes written in C++. C++ is imple- 
mented, currently, as a preprocessor for C. 

• Classes separate operating system mechanism from policy and allow reuse of 
modules. 

• The classes used in Choices may be specialized or modified to create new operating 
system features without jeopardizing the architectural integrity of the system and 
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should encourage advanced operating system research. 

• The systems programming language C++ has not been extended or modified; all 
process, exception and communication mechanisms are written using classes. This 
encourages portability. 

• Hardware and application specific features are encapsulated in classes and separated 
from device independent and application independent code. 

• Many operating system services execute in application space, reducing the size of 
the Choices kernel. 

• The system and its applications use UNIX loader formats and can be built under 
UNIX. 

The system is intended to support future investigations of real-time software organ- 
ization, fault-tolerance, networked computers, and load balancing. Much of the design 
of Choices can be translated into Ada® or other non object-oriented languages. This 
would permit “high-quality production” implementations of the code. However, for the 
purposes of this research, C++ has been excellent. The subclassing and generic func- 
tions of C++ have many advantages in prototyping and maintaining code consistency. 
C++ produces good, fast code, it aids and speeds recompiling and software reuse, and it 
has been ported to a large number of machines. It is available at a minimal cost for a 
license to research organizations. The source of the compiler, linkers, and other utilities 
are available. At this point in time, C++ has many advantages for our experimental 
operating system work. 

Choices is discussed further in Appendix B. The prototype code for Choices (as of 
May 21, 1987) can be found in Appendix C. 


3. CLASP 

CLASP, provides a new implementation of the traditional process model. It allows 
portions of the process to execute on the most appropriate processor architecture. 
CLASP isolates a practical level of homogeneity necessary to implement this sharing; it 
also mitigates dissimilarities between the processor architectures — such as register sets 
and stack frame formats. 

CLASP makes the address space of a single process available to heterogeneous 
CPUs with potentially different instruction sets and performance characteristics. Where 
other approaches have concentrated on enhancing addressing to include the concept of 
remote addresses, CLASP makes a single address space accessible to multiple heterogene- 
ous CPUs. A novel aspect of the CLASP architecture is the inclusion of instructions for 
different processor architectures within the same address space. The CLASP system 
introduces a new construct, the Cross Architecture Procedure Call, to transfer a 
process’s control thread between CPUs. The Cross Architecture Procedure Call — or 
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CAPC — uses each CPU’s subroutine call and return instructions to implement control 
transfers between CPUs. This control transfer mechanism and the shared address space 
make CAPCs more transparent than Remote Procedure Calls (RPCs), which require spe- 
cial stub routines and system calls to implement control transfers between CPUs. 

To use the CLASP architecture, a special CLASP loader links separately compiled 
routines. The CLASP loader recognizes the different object formats for various proces- 
sor architectures and resolves the cross-architecture references. It provides the operating 
system kernel with the information necessary to detect control transfers (e.g., procedure 
and function calls) that cross architecture boundaries. Routines to execute on specific 
architectures are compiled for those architectures. Some frequently called routines (e.g., 
sqrt()) are replicated. Duplicate copies of these routines, each compiled for a different 
architecture, are loaded into the executable file. Calls to any of these routines can be 
directed to the local instance of that routine, saving the network overhead of a remote 
call. This replication is a loader operation. 

CLASP subroutine libraries may contain routines for several architectures. Specific 
routines within a library can be compiled for the most appropriate processor architec- 
tures. A library of subroutines to manipulate large arrays may contain code for several 
architectures; for example, routines that manipulate the array may be compiled for high 
performance vector architectures, such as that provided by the Convex C-l. Other rou- 
tines in the library, which do not perform large calculations, may be compiled for the 
workstation architecture . 1 

Trees, lists, and other pointer-based data structures are difficult and sometimes 
impractical to implement in distributed computing models without a shared address 
space. The SUN Remote Procedure Call dereferences pointers to pass individual ele- 
ments of a pointer-based structure. Pointer dereferencing is adequate for situations 
where single structures are passed by pointer instead of value. Nelson advocates the use 
of subroutines to encapsulate access to pointer-based structures. This approach implies 
changing (or deliberately designing) the applications program to encapsulate accesses to 
these structures. The CLASP software architecture addresses this problem by ensuring 
that the context for a pointer (i.e., its address space) is in effect on the remote processor. 
Applications may use pointers as handles to objects and for true pointer-based struc- 
tures without concern about where a procedure is implemented. 

Many RPC implementations package the entire argument list and send it to the 
remote host. Datagram based RPC implementations send the entire argument list to the 
server in a single packet. Therefore, the argument list must be small enough to fit into a 
single packet. Some implementations provide larger argument lists by supporting 
stream based connections. CLASP supports arbitrary sized argument lists. CLASP uses 


1 These routines also might be compiled for both client and server architectures. Calls to the replicat- 
ed routine can be directed to the local instance of that routine and avoid the overhead of a network tran- 
saction. As was pointed out in the above paragraph, the loader performs this replication and resolves 
references to send most calls to that routine to a local instance of the routine. 
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demand paging to move arguments and data to the server only on request for access by 
the remote procedure. As an example, binary searches through large sorted arrays can 
be efficient because only the accessed portions of the array are transferred to the remote 
processor. Pages, once transferred to the server, remain on the server until they are 
required by the client processor. By leaving the pages on the server, most data eventu- 
ally becomes local to a particular computer system. Pages used only by the client 
remain on the client; pages used only by the server will be transferred to and remain on 
the server. Pages of data used by both processors will migrate between hosts as needed. 

Although CLASP appears to be an approach to distributed computing, it is actually 
an extension of the traditional single-system model onto a new underlying implementa- 
tion for greater performance and ease of use. CLASP mimics a single processor model 
but allows the most appropriate CPU to process appropriate parts of the problem. The 
application program is neither restructured nor recompiled. The choice of which proces- 
sor performs a specific routine affects only the processing rate for that procedure. The 
choice does not alter the semantics for that procedure nor its interactions with other 
procedures in the address space. 

CLASP has been implemented between SUN 3 systems under UNIX. In the next 
year, CLASP will be implemented in Choices. Appendix D contains Ray Essick’s Ph.D. 
thesis on CLASP which details the work done in the last six months. 


4. Path Pascal Release 

A new release of Path Pascal for Berkeley UNIX BSD 4.3 was made this Spring. 
The release was prompted by a number of requests for Path Pascal for SUN worksta- 
tions. The new release corrected a number of bugs in the BSD 4.2 version. The new 
release of Path Pascal has been used for the operating system class at the University of 
Illinois. The new release has been distributed to five sites including the Electrical 
Engineering Department at Cornell where it was used in a network simulation class. 
The new release can be obtained on request from Professor Campbell. 


5. The Choices Interface Compiler 

In Choices, there are a large number of operations that may be conceived of as 
being wrapped around user-described operations. For example, a call to a persistent 
object involves remapping the address space as part of the call and (possibly) again as 
part of the return. Parameter transmission across this interface in some cases (for exam- 
ple, when the object is remote) is not straightforward. 

There are many other examples of this type of "wrapped" interface. The implemen- 
tor of an object may well want to impose a synchronization discipline upon its callers, as 
in the Open Path Expressions used with Path Pascal. This is also best described as 
actions to be taken before and after executing the called procedure; in this case, the 
actions are the appropriate operations on synchronization objects (such as semaphores or 
events). 
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A remote procedure call is a more complex example of the same sort of "wrapped" 
interface. The action that takes place before a call is to prepare a description of the 
called procedure and the parameters for transmission to the server; the action on return 
is to get the description of return value and result parameters from the server and for- 
mat them correctly for the caller again. The server’s logic to handle the specific remote 
procedure call is also a "wrapped" interface with a similar flavor. 

There are many other examples, such as preserving atomicity, journalization of 
input and output letters in a transaction processing system, and establishing commit- 
ment points in a database manager, that are all examples of "wrapped" interfaces. The 
common factors for all of these are: 

1 The procedure to be executed consists of taking some action, eventually cal- 
ling another procedure with the same calling sequence as the first, and then 
taking some other action before returning. 

2 The procedure can be generated from a knowledge of its calling sequence, 
without needing to be adapted to the specific application. For example, all 
calls to an operating system kernel can be translated to the corresponding 
trap operations without knowledge of the function of any particular call. 

If both of these requirements are met, the procedure is a candidate for interface 
compiling. 

The Interface Compiler 

The interface compiler is a program that has, as input, a description of the object 
or objects to be adapted (a C++ tin elude file), and a description of how to generate 
the type of interface required (kernel calls, remote procedure calls, or whatever). It pro- 
duces any number of source files as output; these files give the code needed to implement 
the interface. Multiple source files are often required because the interface may need to 
be compiled into multiple object modules; for instance, the server and client ends of a 
remote procedure call. 


6. Summary 

Based on our prior research and in cooperation with Ed Foudriat, Project EOS has 
built Choices, a prototype experimental embedded real-time system for parallel and net- 
work computer architectures. The system has been implemented on a 10 node Encore 
Multimax. The organization of the software is novel and demonstrates that operating 
systems can be constructed using an object-oriented methodology. The current system 
includes parallel execution of Threads on the Multimax, implementation of Spaces, and 
handling of exceptions and interrupts. Future work will extend this operating system 
with servers, real-time features, and networking. 

Many of the networking aspects of Choices has been prototyped in the CLASP sys- 
tem. This architecture and software system allows a virtual memory to be shared 
between several processors. An implementation of CLASP was built for the SUN 3 
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workstation. 

A new release of Path Pascal was created for Berkeley UNIX BSD 4.3 because of 
popular demand. 

Finally, progress is being made in the design and development of a Choices inter- 
face compiler that will aid the construction of network servers, debugging tools, and 
other utilities and services. 
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(Class Hierarchical Open Interface for Custom Embedded Systems 1 ) 

Roy Campbell, Gary Johnston, Vincent Russo 

University of Dlinois at Urbana-Champaign 
Department of Computer Science, 1304 W. Springfield Ave., Urbana, IL 61801-2987 


1. Introduction 

This paper describes the design for an operating system family called Choices being built for 
the Embedded Operating System (EOS) project at the University of Illinois at Urbana- 
Champaign. Choices embodies the notion of customized operating systems that are tailored for 
particular hardware configurations and for particular applications. Within one large computing 
system, many different specialized application servers may be integrated to form a general pur- 
pose computing environment. We have implemented a Choices Kernel on an Encore Multimax. 

Choices, a Class Hierarchical Open Interface for Custom Embedded Systems, provides a 
foundation upon which to construct sophisticated scientific and experimental software. Unlike 
more conventional operating systems, Choices is intended to exploit very large multi-processors 
interconnected by shared memory or high-speed networks. Uses include applications where 
high-performance is essential like data reduction or real-time control. It provides a set of 
software classes that may be used to build specialized software components for particular appli- 
cations. Choices uses a class hierarchy and inheritance to represent the notion of a family of 
operating systems and to allow the proper abstraction for deriving and building new instances of 
a Choices system. At the basis of the class hierarchy are multiprocessing and communication 
objects that unite diverse specialized instances of the operating system in particular computing 
environments. 

The operating system was developed as a result of studying the problems of building adap- 
tive real-time embedded operating systems for the scientific missions of NASA. Major design 
objectives are to facilitate the construction of specialized computer systems, to allow the study of 
advanced operating system features, and to support parallelism on shared memory and 
networked multiprocessor machines. Example specialized computer systems include support for 
robotics applications, network controllers, aerospace applications, high-performance numerical 
computations, parallel language processor servers for IFP[13], Prolog, Smalltalk, and 
reconfigurable systems. Examples of advanced operating system features include fault-tolerance 
in asynchronous systems, real-time fault-tolerant features, load balancing and coordination of 
very large numbers of processes, atomic transactions and protection. Example hardware archi- 
tectures include shared memory multiprocessors like the Encore Multimax and networked com- 
puters like the Intel Hypercube. 

Choices was designed to address the following specific issues: the software architecture for 
parallel operating systems; the achievement of high-performance and real-time operation; the 
simplification and improved performance of interprocess communications; the isolation of 
mechanisms from one another and the separation of mechanisms from policy decisions. 


1 Thi* work was supported in part by NASA under grant no. NSG1471 and by AT&T METRONET. 



Of particular concern during the development of the system, was whether the class 
hierarchical approach would support the construction of entire operating systems. C++ was 
chosen because it supported classes while imposing negligible performance overhead at run-time. 
In particular, we decided to construct all parallel and synchronization features using C++ classes 
rather than by introducing new language primitives. Thus Choices is also a study of the ade- 
quacy of class hierarchies to abstract and support parallelism and other operating system con- 
cepts and to allow specializations of classes that facilitate efficient support for applications. 

Fortunately for the designers of Choices, there has been a lot of operating system develop- 
ment that is directly applicable to our goals. However, this development work often produced 
implementations buried within the bowels of large, successful operating systems. Abstracting the 
ideas from many different systems and reorganizing them into Choices has been a major concern 
of our design team. 

Choices has been influenced considerably by UNIX® and MULTICS. Indeed, many of the 
standard UNIX system compilers, linkers and utility programs have been used to produce the 
Choices software. However, to structure Choices to allow multiple processes running simultane- 
ously on a multiprocessor with a high degree of parallelism and communication, we have had to 
abandon the UNIX organization of the kernel. Similarly, the UNIX process supports the sequen- 
tial execution of a program running within its own address space. To support real-time and 
high-performance applications, we have opted for a lightweight process. Multiple lightweight 
processes can run on multiple processors within the same virtual address space. Communication 
performance in UNIX is limited by coroutining within the kernel and by copying information into 
and out of user space. In Choices, we have attempted to eliminate these bottlenecks. 

The open architecture of Choices is influenced by the ideas used to build CEDAR [16]. The 
notion of a lightweight process is very similar in Choices and CEDAR, although in Choices it is 
provided through a class abstraction and is not built into the systems language. The Choices 
notion of a lightweight process may be specialized through the subclassing mechanism and this is 
used in the software to distinguish user and system processes. Choices permits a virtual address 
space to be shared by multiple processors. It offers concurrent applications protection from one 
another and hence is, in CEDAR terminology, a closed operating system. However, user created 
operating system policies and mechanisms (like a file system) are provided by the open interface 
of Choices that is supported through the notion of persistent objects. CEDAR is not completely 
built as an object-oriented system although the MESA language is oriented towards encapsulated 
data structures which influences the organization of the system. 

Many current operating system designs address the problem of distributed computing. 
Choices owes several of these systems many of its ideas, but the support of applications on paral- 
lel processors has caused us to implement these ideas in different ways. Many 
distributed/multiprocessor UNIXs (UNIX United [3], LOCUS [10], Mach [1], RFS [13], RIDE [9], 
NFS [18], Encore Multimax UNIX (UMAX) [7], Sequent Balance 8000 UNIX [14]) still impose 
UNIX limitations on the parallelism and performance of applications. Multiprogramming on a 
cached multiprocessor can have undesirable side effects in the form of additional caching and 
cache flushing overhead. Message-oriented kernels like the V System kernel [6], Accent, Amoeba 
[17], and Micros [19] build specific communication schemes into the lowest structures in the ker- 
nel, restricting the possibilities of specializing kernel features to take advantage of communica- 
tion patterns of the application or communication mechanisms of the hardware. For example, 

® UNIX is a Registered Trademark of AT&T. 
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systems implement a few ways of providing “virtual” messages like “fetch on access.” However, 
these systems are not easy to adapt to support other approaches like “send on write”, “send on 
execute”, and “remote procedure call on execute.” Many systems suffer overhead from copying 
messages into and out of virtual memory. Cached systems may pay a double overhead. 

Real-time interrupts and global multiprocessor interrupts pose organizational problems in 
traditional operating system architectures like UNIX. Most operating systems do not include 
parallel programming primitives (for example, the parallel creation of parallel processes), nor can 
they be built easily out of the primitives that exist in such systems. Error recovery is difficult to 
provide in current operating system architectures when used for parallel processing without res- 
tricting parallelism because atomicity constraints cannot be imposed easily. 

The Clouds operating system [2] includes many concepts that have been useful in developing 
Choices. In particular, its notion of a user process accessing a user object is similar to processes 
accessing persistent objects in Choices. Choices differs in not supplying a kernel level atomic 
transaction. 

One of the goals of Choices is to permit the custom design of operating systems for specific 
hardware and applications. General purpose operating systems employ delayed bindings within 
their architectures to provide flexibility. Examples include communication schemes, file systems 
and additional kernel code to handle different architectures and configurations. Choices, on the 
other hand, is aimed at providing the smallest operating system that will support a particular 
application on a particular hardware. Where several applications need to coexist within the same 
computing system, Choices allows these applications to each run on their own custom-built 
Choices operating system. Any communication required between the applications is supported 
by common Choices primitives and shared persistent objects. 

The design of Choices is based upon several assumptions: 

• Embedded, real-time, and server computing services will be provided by large numbers of 
fast processors connected together by shared memory or by a fast network. 

• A computational facility is multitasked (it supports several concurrent applications), where 
each task may use multiple processors. 

• Processes in an application have a high degree of communication. 

• Each application may need to intercommunicate with other applications. Applications com- 
municate less frequently than lightweight processes within a particular application. 

• Communication overheads are small but significant. 

• Even though hardware technology will provide large multiprocessors with very fast proces- 
sors, performance of the applications will remain a critical issue. 

• Small lightweight operating systems are desirable in real-time and high-performance appli- 
cations. 

• Processors within a multiprocessor may be dynamically partitioned to execute different 
applications. 

• Each application may need basic support and specialized support from the operating sys- 
tem. 

• The hardware will support very large virtual address spaces. 

Choices is designed to support specialized applications like embedded real-time systems, 
numerical programs and specialized computing environments like FP or parallel logic programs. 
A Choices system could be embedded as a node within a network of workstations. 


In the subsequent sections, we discuss the class hierarchical organization of Choices and the 
various classes we have built to implement virtual memory, the concept of process, the notion of 
a persistent object and exception handling. 

2. The Choices Class Hierarchy Model 

Several problems emerge when designing an extensible family of operating systems where 
each member can be specialized or customized for a particular application or hardware 
configuration. Each module within the system may have many different versions tailored for 
each different member of the family of operating systems. However, since the different versions 
of a module for different machines or applications all perform a similar function, large portions of 
different versions of a module will be identical. Customizing an operating system for a new 
application requires access to particular aspects of the code that may reside in many different 
modules. 

A class hierarchy provides an ideal solution to these problems. Particular instances of 
classes in the hierarchy are chosen and combined to produce a customized operating system for a 
specific architecture and application. Class inheritance provides for code re-use and enforcement 
of common interfaces. Customization of the operating system for new applications is guided and 
aided by the structure induced upon the system by the class hierarchy. 

A class hierarchy gives more than ease of customization. It also gives us a conceptual view 
of how portions of an operating system interrelate. It is easier to understand and more flexible 
than traditional layered approaches to operating system design. A class hierarchy allows concep- 
tual "chunking" of knowledge about portions of a system by learning the function of parent 
classes and inferring functionality about subclasses. Traditional layered approaches conceptually 
group large sections of functionality into a layer, but the interrelations of the layers are often 
complex and poorly understood. Also changing a piece of a layer is in no way facilitated by the 
layering. However, in a well designed Class Hierarchical model only the top few classes would 
need to be mastered to achieve a good overall view of the system. Class derivation gives a 
method to change specific parts without adversely effecting the whole structure. 

The Choices support for applications is divided into two portions. The Germ is a set of 
classes that encapsulates the major hardware dependencies of Choices and provides an “ideal- 
ized” hardware architecture to the rest of the classes in the hierarchy. It provides the mechan- 
isms for managing and maintaining the physical resources of the computer. A Kernel is a collec- 
tion of classes that supports the execution of applications and implements resource allocation pol- 
icies using the Germ mechanisms. 

Individual customized systems will consist of derived classes from the Germ classes defined 
by Choices appropriate for the particular hardware of the system, plus the specifically tailored 
Kernel classes the system builder desires. Once this hierarchy is laid down, individual applica- 
tions that run on top of the new Kernel can further augment the class hierarchy with their own 
classes. 

In the following sections, we will describe some of the classes that constitute Choices. The 
first set of classes we will discuss provides an abstraction for physical and virtual memory. 

3. Stores and Spaces 

Stores and Spaces are classes of objects which the Choices Germ provides for memory 
management. A Store object encapsulates the management of physical memory. An instance of 
a Store manages a range of contiguous physical memory addresses. Operations are provided for 
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Store instantiation, Store destruction, page allocation, and page deallocation. One application of 
multiple Store objects is to manage memories with different properties, for example, local 
memory, shared memory within a multiprocessor, or global memory shared between multiproces- 
sors. 

A Space object encapsulates the management of virtual memory. An instance of a Space 
manages a range of contiguous virtual memory addresses. Operations are provided for Space 
instantiation, Space destruction, allocation and deallocation of page table entries, changing pro- 
tection flags on page table entries, mapping a page table entry to a physical page of memory 
within a Store, and mapping virtual memory addressing faults on specific page table entries to 
appropriate exception handlers (see the section on Exception Handling in Choices.) 

Many non-overlapping Spaces may be mapped into the virtual address range of a processor 
at any one time. The aggregate of the Spaces addressable in a processors virtual memory is 
represented and managed by an instance of the Universe class. 

Spaces implement protection of the virtual memory they contain by means of the available 
virtual memory hardware protection mechanisms. Protection ensures that a process can only 
access a Space according to the access rights it possesses for that Space. A process may have 
rights to access a Space as a Primitive Space (in the case that it contains a process stack, code, or 
local data) or as a Derived Space containing persistent objects. Primitive Spaces are protected 
from invalid read, write, or execute access. A Derived Space can only be accessed by the methods 
of the persistent objects that it contains. 1 2 The next section discusses the Choices concept^ of a 
process. 

4. The Choices Process Concept: Threads 

Choices is designed to support real-time multiprocessing and parallel computing on large 
numbers of processors. The Choices system supports the concept of a computation that is com- 
posed of a potentially large number of lightweight parallel processes termed Threads. Each 
Thread represents a small independent sequential computation. 

Interrupt and real-time processing requires the ability to switch between Threads with a 
minimum of context switching overhead. A Thread is implemented with a stack pointer, a pro- 
gram counter and a set of register contents. As the Thread executes, it will access its stack, code, 
and data from addresses within various Spaces. To accommodate real-time and interrupt pro- 
cessing, Spaces may lock their pages to be resident in physical memory. In addition, a Universe 
may lock a Space to be resident in virtual memory. A context switch to a Thread that executes 
and addresses only resident pages in resident virtual memory requires minimal overhead. Inter- 
rupt handlers and real-time processes can be implemented in this manner, if desired. Such 
processes may be protected from other applications by setting the memory protection of the 
Spaces they access to exclude access in user mode and by running the processes in the supervisor 
state of the processor. Most Threads, however, will need to access addresses that are not always 
resident in memory or in the Universe. Switching between Threads of this type will usually 
involve at least a partial virtual memory context switch. 


1 A Derived Space is created from a Primitive Space by granting processes access rights to the methods of the objects within the 

Space. In Choices, such objects are called persistent because their existence becomes independent of the lifetime of any one process (see 
the section on Persistent Objects.) We emphasise the distinction between a Derived Space and a persistent object. Although a 
Derived Space can contain persistent objects, the Space itself is a Germ object. 
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A real-time application may use multiple communicating Threads to achieve concurrency 
and parallelism. A Task is a collection of Threads that have common sets of Spaces to minimize 
context switching. Little or no virtual memory context switching or memory cache flushing 
should be needed to switch between two Threads in a Task. Kernel scheduling algorithms can 
exploit Tasks to achieve high performance. Tasks also provide a framework within which 
Thread execution may be prioritized; perhaps to optimize the execution of a parallel processing 
user application. 

A Space Access List is maintained by the Germ for each Thread. This list specifies the 
Spaces a Thread must access in order to execute, as well as the access rights that a Thread has to 
those Spaces. The protection specified by the Space Access List is implemented by a combination 
of hardware and software. For Primitive Spaces, read, write, and execute protection is provided 
directly by the Space through the page table and the paging hardware and memory management 
unit. For Derived Spaces, the protection is achieved by providing “gated” procedure calls to the 
methods of the objects in the Space. As the Thread invokes a gated call, the call is validated, 
and if valid, the Space Access List of the Thread is updated to reflect the protection domain of 
the persistent object. On return, the Space Access List is restored to reflect the original protec- 
tion domain. The Germ supports efficient operations on the Space Access List that are similar to 
the rules in a capability model [15, 20]. 

Communication can be achieved by means of shared Spaces. Popular shared memory and 
message passing communication schemes exist in the system as part of the operating system pro- 
vided class hierarchy. Other user defined communication schemes can be built by extending the 
class hierarchy. An interface compiler for C++ enriches the possible communication schemes. 
Currently, we have included a Path C++ class (named after Path Pascal [4]), monitors, sema- 
phores, messages, and simple varieties of guarded commands. 

Protected communication can be achieved by means of shared Derived Spaces containing 
persistent objects. The methods of such objects may enforce particular communication protocols 
upon the Threads that use them and the protection provided the objects prevents misuse. 

Since a Thread may execute in any Space, a persistent object may include a Thread and be 
active. Active objects can be used to implement name servers and to send asynchronous mes- 
sages. Several persistent system objects augment the shared persistent objects and provide 
high-performance communication channels between Threads and between Threads and devices. 
System objects are implemented in the Kernel or Germ. They can support stream-based com- 
munications, broadcasts, multicasts, and block I/O. Persistent objects are discussed further in 
the next section. 

5* Persistent Objects 

Choices is designed with the objective of placing many operating system and subsystem 
components in a protected Space rather than in a kernel as is done in traditional systems. This is 
done to reduce the interdependences among operating system components and to increase the 
coherence of the components themselves. Such components are implemented as Choices per- 
sistent objects . That is, instances of classes that reside in memory for periods that exceed the 
execution of a particular Thread and that may be shared between multiple Threads. Persistent 
objects may be mapped into the virtual memory of several processors at the same time. In a 
sense, the Germ is a collection of persistent objects that are always resident and accessible in the 
address space of every processor. 
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A full description of the protection scheme used in Choices is beyond the scope of this short 
paper. However, we must introduce enough of the scheme here in order to describe access to and 
the invocation of a persistent object. Each Thread executes within a protection domain that dic- 
tates what the Thread may access. The protection domain of a Thread is dynamic and may 
change by adding or removing Spaces. Initially, the protection domain depends upon the protec- 
tion of the executable file that the Thread is created from and the protection domain of the 
parent Thread. A Thread that executes a method of a persistent object enters a new protection 
domain that depends upon the protection of the Derived Space and the protection domain of the 
Thread. When the Thread returns from the method invocation, its previous protection domain is 
restored. 

F or example, policy modules of the operating system that traditionally are part of the ker- 
nel, may be implemented as persistent objects. A Thread executing one of the methods within 
these persistent objects may require access to Germ data structures. This is possible by having 
the Thread enter Supervisor state to execute the method. The gate mechanism changes the pro- 
tection domain by altering the execution level. 

Threads access persistent objects using an object descriptor and method. A Thread must 
obtain the object descriptor before use. Object descriptors are provided from user or system 
name servers. 

Name servers are persistent objects. Choices includes “standard name servers” that are in 
the Kernel and may be accessed by every Thread. These name servers provide basic facilities like 
the standard file system and intertask communication. Other user defined name servers must be 
accessed through the standard name server utilities. 

On request, the name server grants the Thread access to the object and returns the object 
descriptor. The grant operation is implemented in the Germ and checks Kernel protection policy 
to determine if the name server /Thread grant operation is valid. The name server must have 
appropriate access rights to the persistent object. If the operation is valid, the Germ adds the 
Space of the persistent object to the Space Access List of the Thread, updates the Thread’s 
Universe, and returns the Space address and gate information to the name server. The name 
server packages an object descriptor which includes the persistent object, Space and gate infor- 
mation and returns. 

An operation on a persistent object is invoked through a gated request. The Germ ensures 
that the object descriptor and method used by the Thread gated request correspond to the valid 
persistent object address and method entry point within the Space. The Space Access List of the 
Thread is changed to reflect the protection domain requirements of the Space. 

In hardware architectures with limited virtual memory, the gated method of invoking a per- 
sistent object allows many different Spaces to share the same virtual memory address range. The 
Space and the persistent objects it contains can be mapped into and out of the same address 
range on demand 3 . In such implementations, the Space Access List will contain each Space, but 
only one of the Spaces will be present in the Universe at any one time. 


1 In many hardware architectures, a persistent object must be relocated by a link editor to allow it to execute within a specific 
address range. This implies that once it is activated, it cannot be moved to a new address range. 
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6. Exception Handling in Choices 

Exceptions in Choices are managed by the Exception class and its various subclasses. The 
parent class of Exception defines the method, handle , to manage or correct the exception condi- 
tion. Upon an exception condition, the Choices Germ manages the task of converting the machine 
dependent details of exception processing into an invocation of the handle method for the Excep- 
tion object managing the exception. Various subclasses of Exception define the behavior of handle 
in different ways. Some subclasses of Exception are actually container classes which, based on 
other inputs, send the handle message to another Exception object contained within. 

Two subclasses of Exception of interest are Trap and Interrupt The Trap class provides 
Choices with a mechanism for handling traps that a Thread may generate as a direct result of its 
execution. This includes machine traps (that is, divide-by-zero or illegal instruction), virtual 
memory access and protection errors (that is, page faults of various types), and explicit program 
traps (for example, a “system call”). 

The basic function of a Trap handler is to, if possible, service the exception condition within 
the context of the faulting Thread, otherwise to terminate the execution of the faulting Thread. 

Interrupts occur asynchronously and, in general, have nothing to do with the currently exe- 
cuting Thread. In Choices, an Interrupt can be awaited by a Thread (and must be awaited if it is 
not to be missed). The handle method of the Interrupt class saves the details of the interrupted 
Thread and resumes the Thread awaiting the occurrence of the interrupt. The Choices Germ has 
no requirement that all interrupts be handled by the class Interrupt. A Choices kernel implemen- 
tor can choose to have any type of Exception object handle an interrupt. In future work, various 
user-oriented exception schemes will be implemented as classes and by the interface compiler. 
Examples of such schemes can be found in [5]. 

7. Summary 

A Choices Kernel currently runs on a 10 processor Encore Multimax that supports the 
Store/Space/Universe model of memory management as well as the Task/Thread process con- 
cepts. Current effort is devoted towards improvement and further implementation of communi- 
cation and persistent object support. Future plans include an object-oriented file system, an 
advanced interface compiler, and tools for configuring Choices systems. Once Choices is stable, 
the code will be placed in the public domain to promote research into customized operating sys- 
tems. 
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APPENDIX C 


Choices Code 


Owner russo at rn.cs.uiuc.edu 

Name ../h/Object.h 

Account 173 

Site Dept, of Computer Science 

Printer 24/300 

SpoolDate Thu May 28 1 0:00:52 1 987 


JobHeader on 
JamResistance On 
Language printer 
formwidth 132 
formsperpage 2 
outlines on 

IMAGEN Printing System, Version 2.2, Serial #86:2:85 
Page images processed: 26 
Pages printed: 26 

Paper size (width, height) : 

2560, 3328 
Document length: 

60316 bytes 



Mi y 2 6 2 5 : D 5 133 7 


h/Gb;-jci. A ? age 1 


■/* 

* Cb-ject.a: definition of the Object parent jiisi , the parent of all classes, 

* SHeader; Object. h.v 11. 3 97/05/21. 15 49 55 russo Exp 5 

* SLocksr: 5 

* 

* The destructor for Object is made virtual so ail destructors throughout 

* the system will be likewise. This allows collections of objects 

* to be kept and deleted while assuring the proper destructors will be 

* called for each class. It increases the size of every object in the 

* system by the size of a pointer but who cares, memory is cheap 

/* 

k Modification Histroy: 

* SLog: Object. h,v $ 

| * Revision ii.O 37/05/21 15:49:55 russo 

; * Console input and private stores 

* Revision 10.0 97/04/22 07:33:33 russo 

* New Spaces. Universes and CP ? J objects work. Finally* 

I * Revision 9.0 37/04/04 15:05:34 russo 
• * Multiple threads and timer interrupts. 

j * Revision 3.4 97/03/29 16.54:12 russo 

* added dummy mime constructor. 

* Revision 3.3 87/03/29 16:47:50 russo 

* added Object as parent class 

* Revision 9.1 87/03/29 15:35:45 russo 

* initial revision. 

*/ 

ilifndef Object_h 
[•define Object_h 




ORIGINAL PAGE IS 

of poor quality: 


iMay 26 05:35 1907 /h/Assert.h Page 1 


• Assert . h 


Assert ions . 


/* 


SHeader: Assert.h.v 11.0 87/05/21 15:49:16 russo Exp 5 
SLockar: S 


V 


Modification history: 

• Slog-. Assert.h,v$ 

• Revision 11.0 97/05/21 15:49:16 russo 

• Console input and private stores. 

• Revision 10.0 87/04/22 07:33:01 russo 

• New Spaces, Universes and CPU objects work. Finally! 

• Revision 9.0 87/04/04 15:05:03 russo 

• Multiple threads and timer interrupts. 

• Revision 8 0 87/03/29 15:28:30 russo 

• _new and _delete added for memory management. Also, class interrupts work. 

• Revision 70 87/03/25 12:45:45 russo 

• Fault handler hierarchy works, so does interpr ocessor vectored interrupts. 


Revision 1.1 07/02/23 L8:3L:32 

Initial revision 


fifndef Assert_h 
Idefine Assert_h 


johnston 


I ifdef ASSERT 

extern void _Assert( char * exp, char • file, int line ), 

• define Assert;exp) if( exp ) ,• else _Assert< "exp 11 , FILE LINE ) 

const int NOT REACHED = 0 ; 

• else 


•define Assert'exp) 
lendif assert 


i 


lendif Assert__h 



S 1937 


. . / h/ Debug .1 Page L 


May 2 5 3 5 ■ 3 


V 

* Debug.il - Denuggmg stuff 

* SHeader: Debug, h.v 11.3 37/05/2 1 15 49 3 3 russo Exp j 

! * SLocker; 5 

I V 

;/* 

j * Modification history: 

* $ L eg : Debug. h,v $ 


* Revision 11.0 37/05/21 15:49:33 

* Console input and private stores. 

* Revision 13.0 37/04/22 07.33 20 

* Mew Spaces, Universes and CPU 3b]< 

* Revision 9.1 97/04/20 13:42:25 

* changed debug to user CPUPrmtf 

* Revision 9.0 87/04/04 15:35:19 

* Multiple threads and timer : r. t e r r ' 

! * Revision 3 3 37/03/29 15 29-45 

i * _nev and _lelete added for memory 

j * Revision 7.0 97/OJ/2S 1 12:46:3? 

I * Fault handler hierarchy worts, so 

* Revision 1 1 37/02/23 13:31:32 

j * Initial revision 

i V 

i 

j I 1 f nde f Debug_h 
! I define 0»bug_n 


russo 


russo 

icts work , f inally * 
russo 


russo 

,pts. 

russo 

management. Also, class interrupts work 
russo 

does mt er d r ocessor vectored interrupts 
Johnston 



jextern void Print: ( char * , ... ); 

(extern void C?U?rmtf( char *, ) / 

iextern void PanicPrintf; char * . . )/ 

I ifdef DEBUG 

Idefine Debug CPUPrmtf 


| I else DEBUG 

!// This is tnt instead of void to avoid "sorry not implemented" things, 
t inline /* should be void */ int Debug( char *, ... ) ( return (0); } 


I end 1 f DEBUG 
tendif Debug_h 


i 

t 


[ 

i 

I 

I 


'1 
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/ 


VM ■ h 


Virtual memory management { MMU , page tables, virtual addresses, etc.) 


/ 


$ Header : VK.h,v 11.0 87/05/21 
SLockei: 5 


Modification history: 

5Log : VM. h , v $ 

Revision 11.0 37/05/21 15:50:27 

Console input and private stores 


15:50:27 


russo 


russo Exp $ 


Revision 10.0 87/04/22 Q 7 ■. 3 3 5 9 russo 

Mew Spaces, Universes and CPU objects work. Finally' 


Revision 9.7 87/04/21 05:42:03 Johnston 

Make shift arguments unsigned to avoid sign extension. 


Revision 9.4 37/04/20 09:07:26 russo 

added mimes for conversion from addresses to pages and frames 
Revision 9.2 37/04/16 16:13:59 ]ohnston 

Added page/poiater table typedefs and better initialization routine 
decl ar at ions . 

Revision 9 0 87/04/04 15:06-10 russo 

MultipLe threads and timer interrupts. 

Revision 8.0 87/03/29 15:30:16 russo 

_nev and ^delete added for memory management. Also, class interrupts work. 
Revision 7.0 87/03/25 12:46:47 russo 

Fault handler hierarchy works, so does mterprocessor vectored interrupts 


Revision 4.0 97/03/10 14:32:37 gohnstoa 

All new for 1987* 


/ 


lifndef VM h 
Idefine VM^h 


I include ' md_const ants . h" 
linclude 'Debug. h" 

I include "Assert-h" 

linclude 'Object- h" // parent class of VA and PTE (Cor now} 

/* 

** Build inline function returning the value of the named field. 

V 

Idefine FI E LD FUMC ( name ) unsigned name() \ 

{ return < data field name } 


/• 

* Virtual address. 

V 


! 


1 

i 


ORIGINAL PAG£ 15 j 

OF POOR QUALITY 

i 



May 2 5 0 5:05 1337 
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class VA ' 

-in ion t 

unsigned all,- 
struct { 

unsigned offset 
unsigned secondlev-li ndex 
unsigned f ir stLevell ndex 
unsigned reserved 

} f ieid,- 

} data. 


9 . // ? age offset . 

7,- // level 2 index. 

9.- // level 1 index 

5: // RESERVED by hardware. 


unsigned assign* unsigned all ) 

l 

Assert* ( (VA *) sail )->d#ta . field reserved = = 0 )• 

return ( data. all = all ).■ 

I 


publ 1 C : 


unsigned assign* unsigned llix, unsigned 12 :x, unsigned offset 

{ 

Assert* 11 ix < 256 

Assert* 12 is < 123 } 

Assert* offset < 512 j 

uns igned va = 0 ,- 

( v VA * ) Srv%)->data. field, f ir stLevell ndex = 11 is,- 
( (VA * ) 4va)->dati. field. iecondlevelladex = 12 ix.- 
( * VA *) & va ) - >data . f leid . o f f set = offset, 

return ( assign* va ) ) , 

J 


VA t 

) 



{ assign* 

0 )/ } 

VA* 

unsigned va ) 



{ assign* 

v a ) .- } 

VA ( 

VA i V a ) 



{ data f *11 

=? va -data . all .• 

VA* 

unsigned liu, 

unsigned 12 ix, un; 


{ assign* 

Llix, 12ix, offset 

VA* 

chi: * va ) 



{ assign* 

(unsigned) va ); } 

VA ( 

void * va ) 



{ assign* 

* unsigned* va ) ; J 


) 


unsigned operator^* uns igned va ) 

{ return ( assign* va ) ) ,- } 

unsigned operator^* VA va ) 

{ return * data, all = va. data, all ),- > 

operator unsigned* | 

{ return ( data, all ) „■ } • 


FIELDFtWC* f irstLevellndex ) 
FIELOFUNC* sec ondLevel index ) 
FIELD FUHC * offset } 


} ; 


void print f ( ) / 


ORIGIN AC PAGE IS 
OF POOR QUALITY 
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/* 

* Address conversions. 

*/ 

inline unsigned int 
addrToPage* void • addr ) 

{ 

return* ((unsigned int ) addr) >> PAGESHIFT ) ,* 

I 

inline void • 

pageToAddr* unsigned int pageMumber ) 

t 

return* (void *) ( pageNumber << PAGESHIFT i ), 

} 

inline ua signed int 
addrToFrame* void • addr ) 

{ 

return* ((unsigned int) addr) >> 16 


inline void * 

frameToAddr* unsigned int frameN umber ) 

return* (void *) ( frameH umber << 16 ) 

I 


* Page rounding. 


overload PageFloor, 

inline unsigned int 
PageFloor* unsigned a ) 

{ 

return ( n £. ~ * PAGESI ZE - 1) ) ■ 

} 

inline unsigned int 
Pagerioort void * a ) 

{ 

return ( PageFloor* (unsigned) a ) ) ,* 

} 

overload P ageCe ll ing 

inline unsigned int 
PageCeiling* unsigned n ) 

{ 

unsigned f = PageFloor* n ),- 

return ( (n -= f) ’ n : ( f + PAGESIZE) ) ; 

} 

inline unsigned int 
PageCeiling* void • a ) 
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/ h ‘ VM h Page \ 


return < PageCeiUng, 


* Page table entry. 

I V 

j const unsigned MAX HAND LESS - IS. 
l 

| class PTE { 
j union { 

| unsigned all; 

struct 


; data; 
PTEh 


// Only 4 bits to vorfc with in the PTE. 


uns igned 

valid 

1 . 

// 

Val id . [ 

uns igned 

protect lonLevel : 

2 . 

// 

Protection level. 

uns igned 

referenced 

1 ; 

// 

Re fer enced . 

uns igned 

modified 

1 

// 

Modified. 1 

uns igned 

handler! r.dex 

4 

// 

F aul t handler index . | 

uns igned 

pageNumber 

23 

// 

Page frame number. 


data . all = 
PTE ( PTE t pte ) 


data all = pte.data.all; 
data . field, referenced = 0/ 
data . f xeld . ntodif led = 0 ,- 


PTE; unsigned pte 


data all = pte,- 

data. field referenced = 0 . 

data field modified = C; 


;• 


unsigned operators; PTE pte ) 

data. all = pte.data.all,- 
data . field . ref erenced = 0/ 
data . f ield . mod i f led = 0,- 
return ! data . all ■ 


{ 


unsigned operator 1 : unsigned pte ) 

data all - pte, 
data . f ield re f erenced = 3,- 
data . field modif led = 0,- 
return ( data, all } ,- 
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j 


operator unsigned; ) 

{ return ( data. all } 

void map; unsigned pa, unsigned pi ) 

{ 

Assert; pn < 0x3000 ) .- 
Assert; pi < 0x4 
Assert; ! dat a . f leid . val id 
data, field pageNumber = pn.- 
dat a . f ield pr otec t lonLe vel 
data . field, val id = 1,- 
data . f ield . re ferenc ed = 0; 
data . field. modif led = 0 

) 

void map; unsigned pn ) 

t 

Assert; pn < 0x8000 ) ,- 

Assert; ' data . f ield . val id ) 
dat a . f ield . pageNumber = pn,- 
data . f ield. val id = 1 ,- 
dat a . f ield . re f erenced = 0/ 
data . field, modif ied = 0 ,- 


vo id unmap; ) 

{ 

Assert; data, field, valid ),- 
dat a . f ield . val id = 0; 

f 

void handle; unsigned hi. unsigned pi ) 

{ 

Assert; hi < MAXHANDLEBS I ; // Handler index m range. 

Assert; pi < 0x4 ) , // Protection level m range 

data . field. handler I ndex = ai, 

data . f ield . protect lonLevel = pi, 

data . f ield. re f erenced * 3,- 

data . f ield . modif ied = C ; 


// Page number in range. 

// Page not already mapped. 


// Page number in range. 

// Protection level in range. 
// Page not already mapped. 

pL ; 


FIE LDFUNC ( 
FIE LDFUNC ( 
F I E LDFUNC ( 
FIELDFONC; 
FI E LDFUNC ( 
FI E LDFUNC ( 


pageNumber },- 
handlerlndex ) ,- 
modified ) ,- 
referenced ),- 
protec tionLevel 
val id ) .- 




! void pr intf ( ) ,- 

! } ; 

/* 

* Page table initial ization routine*. 

V 


j 


i 


typedef PTE P ageT abl e [ 2 56 ] . 
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I 


i-er? ioi«; 123 ; . 


| extern void 

extern PTE * 
extern PTE * 


. -P able PageT able » , 

: mcPo intertable { Pointer? able 


ImtFir st l evelP ageT abl e ? PTE * ); 

InitSecondLevel? ageT able f PTE * ) ; 


/* 

* MMU registers. 
V 


/* 

* Error/ I aval idate Address. 


" &litory ORIGINAL PACE IS 

OR POOR QUALITY 


class E I A | 

union [ 

unsigned all; 
struct { 

unsigned iddress 
unsigned reserved 
unsigned txPTB 

; field. 


unsigned assign? unsigned eia ) 

{ 

return ( data . all = eia ),- 

} 

unsigned assign( unsigned txPTB, unsigned address ) 

{ 

Assert? , txPTB == 0 ) ! | ( txPT3 ss 1; ) ; 

Assert? address < OxlCOOOOO ) : 
unsigned eia * 0 

(EIA *) ieia )->data. field. txPTS = :xPT3. 

■: (EIA *) seia >- >data .; leld . address - address, 
return? assign? eia ) ) 

} 

publ 1 C : 

EIA? ) 

[ assign? 0 ); } 

EIA? unsigned eia ) 

{ assign? eia ); ) 

EIA? EIA & eia ) 

l data. all = eia . data . all .• } 

EIA? unsigned txPTB, unsigned address ) 

{ assign? txPTB. lddress ) } 

unsigned operator*? unsigned eia ) 

{ return? assign? eia ) ),- } 

unsigned operator*? EIA ait ) 

{ return? data, all * eia. data, all ),- } 

operator unsigned? ) 

{ return ? data. all ); ) 


/ / Fault address. 

, ' RESERVED by hardware. 
// PTB0./1 did translate. 
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unsigned address? ; 

{ return ? data . field. address 
unsigned txPTB?) 

{ return ? data . field . txPTB ); 

vo id pr lfttf? ) ; 

) ; 

/* 

* Memory status register. 

V 


[class MSS { 

union { 


unsigned all,- 
struct { 

unsigned txError 
unsigned magic 
unsigned BPTError 
unsigned Prot LsvelError 
unsigned LIPTEError 
unsigned L2PTEEr r or 
unsigned BPS 
unsigned reservedl 
unsigned readError 
unsigned BPTReadError 
unsigned txStatError 
unsigned BPTStatError 
unsigned txffser 
unsigned txSupervisor 
unsigned userPTB 
unsigned override 
unsigned BPTEnable 
unsigned BPTOserOnly 
unsigned ai 
unsigned flovTrace 
unsigned flowJserOnly 
unsigned nonseqTrap 
unsigned reserved2 
f field; 


)/ f 

J 


// Address translation error. 

// Clear MS a. 

// Breakpoint error. 

// Protection level error. 

// First level PTE error. 

// Second level PTE error. 

// 9PR 0/1 caused error. 

// RESERVED by hardware. 

// tfr its. read error 
// Write/read breakpoint error. 
// Translation ous cycle error"* 
// Breakpoint bus cycle error"* 

// Translate user addresses. 
f j Translate supervisor addrs 
// 7se PTB0/1 for user. 

// 'Jse super, prots. for user. 

// Enable breakpoints. 

// 3r eaxpo int in user mode only. 
// Abort/WKI trip’ 

// Enable flow tracing. 

// Flow trace in user mode only. 
// Enable nonseg. flow traps’ 

// RESERVED oy hardware. 


unsigned assign? unsigned msr ) 

t 

Assert? (?MSR *) insr )->data. field, reservedl == 0 ) ,- 
Assert? ((MSR •) imsr j - > dat a . f leld . r -served! *= 0 ).- 
return ( data. all = msr ) ,• 


unsigned assign? unsigned magic, unsigned txUser 

unsigned txSupervisor unsigned userPTB j 

Assert? (magic ”* 3 11 (magic == 1) ),■ 

Assert? (txCJser == 0 j j| ititJser =- 1 ) ); 

Assert? (txSupervisor == 0) |{ txSupervisor 


1 ) ) .- 
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publ ic • 


Assert user??3 == : > , user?T3 == / • 

unsigned nsr = •: ,• 

■(MSB *> imsr )- -’diti . f leld magic = magic,- 
•MSS *) ims: i->dau field txUser = u'-'aet, 

MSB *i imsr)->dati. f :eid tsSupervisor = txSupervisor 
.'MSB *) imsr)->data. field. userPTB = user?T3. 
return) assign) msr ) .- 


{ assigm 3 ) ; | 

MSR; unsigned msr ; 

{ assign) msr ) } 

MSB/ MSR i msr i 

{ data, ill = msr. data all. } 

MSB) unsigned magic., unsigned cxUser 

unsigned txSuper v iso r , unsigned userPTB ) 

{ assign/ magic. tx’Jser txSupervisor. userPTB ) .- 

unsigned operators unsigned msr 5 
[ return, assign, msr >. J 
unsigned operators MSR msr 

{ return) data, all = msr. data, all ) } 

operator unsigned)) 

{ return) data. all ) ,• ■ 


FIELDFUNC) 

txError > 

FIELDFUNC) 

magic 

FIELDFUNC; 

BPTError ) 

FIELDFUNC ( 

ProtLevelError 

FIELDFUNC ( 

L IPTEErr or ) ,• 

F I ELDFUNC ■’ 

L2 PTEErr or l ; 

FIELDFUNC • 

BPS 

FIELDFUNC.' 

reservedl _• 

FIELDFUNC i 

read* rror ) ; 

F I ELDFUNC ‘ 

BPTReadError j 

FIELDFUNC.' 

txStatError j ,■ 

FIELDFUNC. 

BPTStatSrror , 

FIELDFUNC ; 

tx'Jser . ; 

FIELDFUNC) 

txSupervisor } 

FIELDFUNC/ 

userPTB ) ; 

FIELDFUNC.; 

override ) .- 

FIELDFUNC) 

BPTEnable j ,- 

FIELDFUNC) 

BPTUserOnly ) 

FIELDFUNC.' 

a 1 ) . 

FIELDFUNC) 

f lowTrace , ; 

FIELDFUNC) 

flowUserOniy ) 

FIELDFUNC) 

nonseqTr ap ),- 

FIELDFUNC; 

reser?ed2 ) 


vo id pr int f ( ) 


MMU register read/writs routines. 
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extern 

vo id 

WnteEIA) EIA e la 

>; 

// 

extern 

unsigned ReadEIA)); 



extern 

vo id 

WriteMSS; MSR msr 

); 


extern 

unsigned ReadMSH)); 



extern 

vo id 

Wr itePTBQ ( void * 

ptb ); 


extern 

vo id 

wntePTBl) void * 

ptb ) ; 


extern 

vo id 

* ReadPTBO ( ) ; 



extern 

vo id 

* ReadPTBl)),- 



1 end if 

VM_h 
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May 2 5 3 5 0 5 19 3" 


?hy j i c i i memory allocation. 


3 Header Store h , v 
SLocker $ 

11 0 37/05/21 15:50: 

Modification history: 

SLog: Store h v $ 

Revision 11.0 37/05/21 15:50:13 

Console input and private stores. 

russo 

Revision 10.4 37/05/15 

Added another instore. 

06:53:34 

johnston 

Revision 10.2 37/05/15 

Fixed offset problem m 

36:17:59 

the mark 

johnston 
functions . 

Revision 10.1 37/05/15 

Redid Store 

94:43:45 

johnston 

Revision 10.0 37/04/22 

0733:45 

russo 


of ^ Ac PA ® a 

P °OR QUAUTB 


Mew 3pac»i. Universes and CPU objects work. Finally 
Revision 9.1 37/04/15 15 39 34 johnston 

Fixed constructor to use physical memory pages to allocate itself 

Revision. 9.9 37/04/04 15:05:00 russo 
Multiple threads and timer interrupts. 

Revision 3.0 37/03/29 15:30:04 russo 
new and delete added for memory management. Also, class interrupts work. 


’ * Revision 7.0 37/03/25 12:45:30 russo 

: * Fault handler hierarchy works, so does interprocessor vectored interrupts 

* Revision 1 1 37/02/23 13.31:14 johnston 

! * Initial revision 

! V 

t 

lifndef store__h 
^define 3tore_h 

I include "Assert. h" 

[I include "Debug. a" 

I include "Lock h' 

1 I include "Object , h*’ 
i I include " VM . h 

I class Store : public Object { 

I Lock lock; 

] unsigned basePage. 

unsigned highPag-; 
unsigned free? ageCount ; 
j unsigned setEntryCount / 

unsigned pagesPerSetEntry ; 
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I uns igned set ( l ] .• 

j void mark* unsigned page ),- 

| void unmark* unsigned page )/ 

i int marked* unsigned page j 

! unsigned next Free* unsigned lowpage ); 

i unsigned contiguous* unsigned iovPaga, unsigned pageCount ),- 

| public: 

> Store* unsigned basePage, 

( unsigned pageCount, 

unsigned • st ateBasePage ) „■ 

“Store* ) .• 

char • allocate* unsigned pageCount ) / 

void deallocate* char • baseAddr, unsigned pageCount ); 
void reserve* unsigned basePage, unsigned pageCount ) ; 

int instore* unsigned page ),* 
tat instore* char * adds > ,- 


j ini me ut 

| Store : : instore f unsigned page ) 

return * page >= this- >baseP age ) a ; page < this- >h ighPage ) ). 

| } 

, inline mt 

Store :: instore * char • addr t 

return * this-> instore* addrToPage* (char M PageFloor. addr } > ) i; 


i ini iae mt 

Store :: marked* unsigned page ) 

;{ 

Assert* th is- > instore* page ) ); 

unsigned offisetPage * page - this- > basePage; 
j return * this->set( offsetPage / th is- > pagesPerSetEntry 

| < 9x1 << * offsetPage % this- > pagesPerSetEntry 


} 


inline void 

Store: mark* unsigned page ) 

l 

Assert* this-> instore * page ) ) ; 

Assert* 1 th is- >marked ( page ) )/ 

unsigned offsetPage = page - th is- >baseP age ; 

th is- > set [ offsetPage / this- >pagesP«rSet£ntr y ) 1 = 

; Oxi << * OffsetPage \ th is- > pagesPerSetEntry i 

i Assert* this- >marked* page ) ); 

Assert* this- > Creep ageCount '=0 ); 
this- > freeP ageCount-- . 


) ; 
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Store : ur.air< unsigned page ; 

{ 

Assert / th is- > instore ( page 
Assert( th is- >marfced ( page > > „• 

unsigned offsetPage = page - th is- > b aseP age 
this->setC of f setPage / this- > pagesPerSetEntry ] i = 

C Oxl <t ; of f setPige 4 th l s- > p igesPeisetE ntry ) 

Assert( ' this->marfced( page ) ) ; 

Assert ( th is- > freeP ageCouat < ( th is- >h ighP age - th is- > bisePage ) ); 

th l s- > f r eeP ageCount++ ; 

i 

tend if 
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/* 

% md_ tuneable h - Machine-dependent, tuneable parameters 

* SHeader; md_tuneable . h v 11.0 37/05/21 15:50:40 russo Exp $ 

* SLocker. $ 

* Modification history: 

* SLog: rad_tuneable . h , v $ 

* Revision 11.0 37/05/21 1553:40 russo 

* Console input and private stores. 


* Revision 10.0 37/04/22 07:34:03 russo 

* Mew Spaces. Universes and CPU objects work. Finally* 


* Revision 3.0 37/04/04 15:06:19 russo 

* Multiple threads and timer interrupts. 


* Revision 3.0 87/03/29 15:30:25 

* _cev and _deleta added for memory 

* Revision ?,0 37/03/25 12:46:53 

* Fault handler hierarchy works, so 

* Revision 41 37/03/09 12:42:47 

* Initial revision. 


russo 

management. Also, class interrupts work 
russo 

does interprocessor vectored interrupts 
} ohnston 


lifndef md_tuaeabLe_h 
Idefme md_tuneable_h 


* Page frame ranges and addresses. 


* MOTE A ’frame 1 ' is the space mapped by a single * f irst- level * page table 

* entry /54k Dytes, here). 


const int MFRAMES 


2 56, 


// Complete address space :0.,255 
// This is 64k * 256 = - 6M bytes. 
'/ This ‘must* be <s 256. 


const mt 
const int 
const int 


GKLOWFRAME 

GKLOWPAGE 

GKLOWADDR 


0X000000 , 

0x000000 ; 
0x000000 ; 


// Germ/ Kernel (0..i; 


const int GKHIGHFRAME - 
const int GKHIGHPAGE = 
const int CKHIGHADDR = 


0X0000 7 f ; 

GKHIGHFRAME << 7, 

GKHIGHF RAME << 16, 


const int STACK LCWF RAME = 
const int STACKLOWPAGE 
const int STACKLOWADDR = 


GKHIGHFRAME 
GKHIGHPAGE . 
GKHIGHADDR, 


// System stack (127 


const int STAC KHIGH FRAME - 
const int ST AC KHIGH PAGE = 
const mt STACKHICHADDR = 


0x000030 

STACKHIGHFRAME << 7, 

STAC KHIGH FRAME << 16,. 
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'coast ;nt TASXLGWFRAME - 
! coast int TASSLOWPAGE = 
•const int TASKLOWADDS - 

I const mt taskhighframe = 
const inc TASKS I GHP AGE = 
const int TASKHIGHADDR = 


STACKS : GH FRAME 
STACKHI GSPAGE , 

STACKS : GHADDR. 

0x0000 fc 

TASKHIGHFRAME << 
TA3EHIGH FRAME << IS; 


const int THREADSTACKFRAKE * 251. 


[const int HWLOWF SAME = 

I const int SWLOWPAGE = 
[const int aWLOWAODR = 

[const int HWBIGHFftAME = 
; coast int awsiGHPACB’ = 
'const lilt H’vfSI GHADDR = 


TASKHIGHFRAME; 
TASKHI GHPAGE ; 
TASKHIGHADDR; 

0x000100; 

HWHIOH FRAME << 7; 

HWH1GHT SAME << IS; 


// SW -252.255). 


• I end i £ aid tuneable_h 
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mi^tuneabl*. h: Mach in*- independent , tuneable parameters. 

SHeidar: mi_tuneable . h , v 11.0 87/05/21 15:50:43 russo Exp S 
$Locfcer: s’ 

Modification History: 

Slog: ai^tuaeable . h , v $ 

Revision 11.0 87/05/21 15:50:43 russo 

Consol* input and private stores. 

Revision 10.1 87/05/07 05:42:15 johnston 

Changed KAXCPUS from 32 to 64. 

Revision 10 . 0 87/04/22 07:34:11 russo 

Kev Spaces/ Universes and CPtJ objects wort. Finally* 

Revision 9.1 87/04/15 15:45:40 johnston 

Added MAXEERNELS. 

Revision 9.0 97/04/04 15:06:21 russo 

Multiple threads and timer interrupts. 

Revision 80 87/03/29 15:30:27 russo 

_new and _del*te added for memory management. Also, class interrupts work. 
Revision 7.0 87/03/25 12:47:01 russo 

Fault handler hierarchy works, so does interprocessor vectored interrupts. 

Revision 4.1 87/03/09 12:43:02 johnston 

Initial revision. 


Iifndef mi_tuneabl*_h 
! tdef in* mi tuneable h 


| Idef ine KAXCPUS 

jldefiae kaxkernels 


i lendif mi_tuneable_h 
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aid constdnca.ii 


Machine-dependent: constants 


SHeader: md_constant 5 . h v 11.0 37/05/21 15*50.37 russo Exp 3 
Slacker: 5 


Modification history: 

SLog: md_constants h,v 5 

Revision 11.0 97/05/21 15:50:37 russo 

Console input and private scores . 


* Revision 10 . 1 37/05/15 23:35:35 russo 

* added 1A3TADDRES3ABLEL0CAT ION 


Revision 10.0 97/04/22 07:34:05 russo 

New Spaces 'Jmverses and CPCT objects work, Finally* 

Revision 9.0 37/04/04 1 5:05:16 russo 

Multiple threads and timer interrupts. 

Revision 3 0 37/03/29 L5:JQ:21 russo 

_new and ^delete added for memory management. Also, class interrupts work. 
Revision 7 j 37/03/25 12:46:54 russo 

Fault handler hierarchy works, so does inter j roc e s so r vectored interrupts 

Revision 4.1 37/03/09 12:42:25 jOhnston 

Initial revision. 


i 1 i f nde f 

md constants 

Idefme 

tnd_:on 

st ants 

! 

! |de f me 

INI? IA 

LMSR 

; Idef me 

LAS TAD D RES 3 A1 

1 

♦define 

PAGES I ZE 

1 | def ine 

j 

PAGESHIFT 

I 

j Idefme 

PSR CT 

3 IT 9 

I Idef me 

?SR U 

255 

! Idef me 

PSR C 

BIT 0 

j Idefme 

PSR C~ 

1 

I Idef me 

PSR I 

BIT 11 

i Idefme 

PSR I~ 

2048 

J Idefme 

PSR S 

BIT 9 

j Idef ine 

PSR S“ 

512 

Idef me 

PSR 5 

BIT 9 

Idef ine 

PSR 3“ 

312 

i Idef me 

PSR T 

BIT 1 

I Idefme 

PSR T* 

2 

| Idef me 

PSR F 

BIT 5 

i Idef me 

psr~f~ 

32 


0x00070002 


Initial value to Load into the MS R 


/ * Number of bytes per page »/ 

/ * Number of bits for page offset V 


ORIGINAL PAGE IS 
QE POOR QUALITY 


l 


I 




I 


| 

I 


i 
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/* 

* DPC control/ st atus registers. 

V 

tdefine DPC_STATCTS Oxfffffffc 

|de fine DPC_CONTROL 0xfffffe43 


/* DPC status register •/ 
/* DPC control register. */ 


tdefine DPC__STATUSJ/BCTS_BCTSY 0x0000000 


/ * Vector bus busy. */ 


/• 

* DPC Vector Bus registers. 

V 

Ide fine DPC VBCTS_CLASS 0xffffte22 

! Idef me DPCJ/BUS TX 0xfffffe24 


/* 

* Encore stuff -+ 


/• Class register. */ 

/* Transmit register . •/ 


I V 


V 


| 1 define 

IO BASE 

0x800000 

| Ide f ine 

1534 BASE 

0x800060 

j Idefme 

SL BASE 

1534 BASE 

i Idefme 

SYS CLOCK 

(1534 BASE+4) 

♦define 

COCTNTER 

(I534~BASE+6 ) 

j Idefme 

MAXCOCTNTER 

Orff 

] Idef ine 

DSD WAD DR 

OxOObcOO 

♦define 

DSDJNIO 

OxSOObcO 

! Idef ine 

ICCT BASE 

OxfffeOO 

I define 

I L _3AS E 

■3x800000 

Idef ine 

PPI BASE 

OxcO 0 0 2 0 

| Idef ine 

CPU NMI REG 

PPI BASE 

! tdefine 

GENNMI 

OxOd 

| Idef me 

ARMNMI 

0x0c 

j Idef me 

CPCT I? I REG 

[ PPI BASE +2 ) 

; Idefme 

cpct“idreg 

i PP I ~BASE+-4 j 

♦define 

CPCT~I0 

0X7 

♦define 

FPI~CTL 

(PPT BASE+S) 

Idef ine 

PPINIT 

0x8 1~* 

Idef ine 

ROMKILL 

0x0a 

tdefine 

CPCT SWREG 

OrcOO 0 30 

tdefine 

SW BOOTCPCT 

7 

Idef ine 

3W~CL0CK 

6 

Idef me 

SW~NET 

5 

Idef ine 

sw’disk 

4 

tdefine 

SW~SE RIAL 

3 

Idef me 

CPCT MCHECEREG 

OxcO 0 0 3 0 

Idef ine 

CPCT_MPARREG_BASE 0x800040 


/* Base of I/O space */ 

/* Base if 4-line asynch card */ 

/ * Serial line base */ 

/* System clock address •/ 

/ * Free running counter address */ 
/* Maximum value of counter */ 

/* DSD wakeup address */ 

/ * DSD wakeup I/O address */ 

/* Interrupt Control Unit * / ■ 

/* InterLAN NI3010 V 

/ * Base of Parallel Ports */ 

/ * CPCT ami register V 
/ * Generate nmi code */ 

/* Arm nmi code */ 

/* Intercpu interrupt register * / 

/ * CPCT identifier register V 
/ * Masx for cpu id */ 

/* Control register */ 

/ * I nit code * / 

/* Z ill rom code code V 

/* CPCT switch register V 
/* Boot processor */ 

/* Clock interrupt processor V 
/• Network interrupt cpu */ 

/* Disk interrupt cpu */ 

/* Serial line interrupt cpu */ 

/* Machine check register */ 

/* Memory parity register base */ 


t 


I 


tdefine NUMSY3INTRS 256 
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♦ def me 

.TJMST3 

♦define 

3CCMEM 

♦define 

: ?,cau 

♦ def me 

DPCREG 

I ♦def me 

DPCREG 

Ide f me 

DPCREG 

Ide f 

me 

DPCREG 

♦define 

DPCREG 

Ide f me 

DPCCTL 

♦ def me 

DPCCTL 

♦ def me 

DPCCTL 

♦ def 

i ne 

DPC3TS 


♦ define 

♦ def :r.e 

♦ def me 

♦ def ms 

♦ def me 


DPCC?L_GEN_SYSNMI 2097152 
DPCCTL FI X~5 08 3020 8 
DPC3TS_CPUID 1 
DPCSTS'SLCTID 50 
DPCSTS” OUT?UT_READY 258435455 
DPC3TS_YEC3US_TXREQ_BIT 27 
D?CSTS_FIX -535858112 
DPCREG~7ECT0R -512 
DPCREG~FI FQ -472 

D?CSTS~: S3X_P RESENT 107374 132 4 
0?CREG_o3:«:C?LO - 3 19 
DPCRSG~S3XDATAC - 3 L5 
DPCREG~3BXCTLI -111 
DPC3EG_SBXDA7Ai -307 
DPCREG TSEVEC WRITE -455 
DPCRE0JTSEC7L -451 
DPCREG TSECNT1 -459 
DPCREG~TSECNT2 -455 
DPC FIFOSI2E 17 
DPCSBXCTL_TXRDY_BIT 2 
DFCS8XC7L RXRDY BIT 0 


ORIGINAL PAGE IS 
OE POOR QUALITY 


I I end l f md 3oa»tar.ti_h 
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Task h: taste class description 

S Reader : Task.h.v 11.0 87/05/21 15:50:16 russo Exp 5 
SLocker: 5 


* Modification History: 

* $ Log : T ask . h , v $ 

* Revtsion 11.0 87/05/21 15:50:16 russo 

* Console input and private stores. 

* Revision 10.3 87/05/12 15:06:54 russo 

* added mitialThread member function 


* Revision 10.0 87/04/22 07:33:48 russo 

* Mew Spaces, Universes and CPU objects work, Finally 1 

* 

* Revision 9.3 37/04/20 13:25:57 russo 

* added a lock to the instances 

* Revision 3.0 37/04/04 15:06:02 russo 

* Multiple threads and timer interrupts. 

* Revision 8 0 37/Q3/29 15*. 30:08 russo 

* _nev and ^delete added for memory management. Also, class interrupts work. 

* Revision 70 87/03/25 12:46:36 russo 

* Fault handler hierarchy works, so does interprocessor vectored interrupts. 

* Revision 4.1 37/03/09 16:13:10 russo 

* initial revision. 


I if ndef Task_h 
♦define Task h 


♦ include 
I include 
♦ include 
I include 
I include 


Object. h" // parent class' 

FauitKandler . h" 

Lock . h" 

Space . h" 

Thread. h" 


typedef void (* TPFV)(),- 

class Task ; public Object { 

Space v space,- 
Lock lock.- 

r aultHandler • st ackFaul tHandler , 

Thread * threads, 

publ ic . 

T a sk ( space * space, TPFV init lalEntryPo int ) 

“Task ( ) ; 

Thread * mitialThread! ) ; 

Thread * startThread< TPFV entryPoinc. int argument 
Space *• getSpace! ) { return! space ); } 


/h/Thread . h P age 1 



[ * Thread. h: thread class description 

! • $ Header •- Thread. h,v 11.2 87/Q5/2L 22.25:39 russo > 

i * S Locker : $ 

I */ 

\/* 

j • Modification history: 
j * $ Log ; Thread . h, v S 

• Revision 11.2 87/05/21 23:25:39 russo 

• removed unneded method 

• Revision 11.1 87/05/21 23:23:10 russo 

• added instance variable to save the initial USP . 

• Revision 11.0 87/05/21 15:50:19 russo 

• Console input and private stores. 

• 

* Revision 10 13 37/05/13 18:55:39 russo 

* made GermThread a subclass cf Thread, not KernelThread . 

* Revision 10.11 37/05/12 09:50:59 russo 

* added new GermThread subclass 

* Revision 10 10 87/05/10 21:05:28 russo 

* each thread now carries around a smalL interrupt stack of its own 

* Revision 10.0 37/04/22 07:33:51 russo 

I * New Spaces, Universes and CPU objects work, Finally' 

I * Revision 9.0 87/04/04 15:06:05 russo 
| * Multiple threads and timer interrupts. 

* Revision 9.9 3 7/04/01 16- 15:00 russo 

• * added offsets for assembler to use 

i - Revision 8.0 87/03/29 15:3011 russo 

* _nev and _delete added for memory management. Also, class interrupts work. 

* Revision 7.3 37/03/25 12:4641 russo 

* Fault handler aierarchy works, so does interprocessor vectored interrupts. 

* Revision 1.1 37/02/23 18:31:39 jchnston 

* Initial revision 

V 

lifndef Thre ad_h 
(de fine Thread_h 

(include "Object. h" // parent class 

I include " Space . b" 

const int stackSize = {512 - 3 2); 

class Thread : public Object { 
protected : 
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char 


:atacruct:t idciscacicSize; 


publ ic 


char * atackPoiater , 
char * mitialUS?.- 
int priority,- 
void * keraelLafo. 
typedef void f* PFV) < ) ,■ 
Space * lspaces. 


// list of spaces this Thread needs to run 


1 


Thread * next,- // for Uniting the Thread into lists 
Thread * last,- // for linking the Thread into lists 

Thread » PFV, int * , mt, int. void * ) ,• 

“Thread () .- 

char * interrupts tackPo i nter ( ; { returaf stickPo inter ) ,- } 

void setlnterruptStackPoiatexi char * isp ) 

{ stacicPo inter = isp; > 

Jpace * spaces' i ! return; lspaces ; } 

void setScaces- Space * s ) { lspaces = s, \ 

void * getXerneil r.f o' ; ; return; fcernellafo ) ,• } 

mt getPr lor ltyt ) { return; priority i } 
void dump < ) . 

virtual int isPreempt able < ■ 


class KernelThread public Thread { 
protected : 
publ 1 C : 

KernelThread.; pr/, mt *, int int, void * }, 

“Ser aelThre id{ ) 

} . 




Jclass CermThread : public Thread ( 

protected : 
publ ic : 

GermThread( CermThread *, PFV. int *, int. int, void * » . 

'GernThread' ) ; 

i ; 

class I nterruptThread public KernelThread l 
protected : 
publ 1 C : 

InterruptThreadi PFV, int *, mt , mt, void * > ; 

~XnterruptThread( ) ,• 
mt isPrsempt able; ) / 

} 

lendif Thread^h 
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/ 


CPU.h-. per-cpu private information class 

53eader: CPU.h,v H.l 37/0 5/31 16:41:50 russo 
Stacker : 5 


Modification History: 

5 Log : CPU h,v S 

Revision 11.1 37/05/21 16:41:50 russo 

only need a single thread to delete not a queue. 


Exp $ 


Revision 11.0 37/05/21 15:49:29 russo 

Console input and private stores. 

Revision 10.30 37/05/17 13:57:09 russo 

added DeleteQueue member 

Revision 10.27 37/05/11 17:39:37 russo 

added local and global store fields. 

Revision 10-22 37/05/06 16:52:47 russo 

made number of vectored exceptions a member function 

Revision 10.21 37/05/04 19:09:27 russo 

added td< ) method 


Revision 10 18 87/04/27 19:14:11 russo 

added Number of vectored exceptions const 

Revision 10 10 37/04/26 21.22.29 russo 

added Exception stuff 


• Revision 10.0 37/04/22 07:33:17 russo 

• New spaces, Universes and CPtJ objects work. Finally 

• Revision 9.9 87/04/21 15:11:57 russo 

• added interrupt stack member. 

• really V irtualSetOp should be a friend to save a lot of set function 

• that should never be called by anyone else. 

• Revision 9.1 87/04/21 09:53:21 russo 

• initial revision 
V 

lifndef CPU_h 
Idefme CPU_h 

♦include "mi_tuneable . h" 

♦include " Space. h" 

I include '‘Thread, h" 

♦ include " Umver se h *’ 

♦include "Exception- h" 

♦ include '* Scheduler h‘* 


t 


i 


! 
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| cons 

! cliss CPU 
| prote< 


/' Really 255 - 15 but. 
-at nExcections = 20, 


’-t ana 11 for now 


publ : z Ob] ect 


ted: 

Univer se 

Thread 

Thread 

Space 

Except ion 

Store 

Store 

Scheduler 

Thread 


j publ 1C : 


rputfniverse ,- 
cpuCurrentThre ad. 
cpuIdleThread, // I ’ 

cpuHeapSpace.- 
cpuThiags[tExcepcions] , 
cpuPrivateStore,- 
cpuGiobalStore,- 
cpuScheduler, 
c puThreadTo Delete ; 


not sure about this 


CPUf CPU 
“CP(J( J , 


* access functions 

*/ 

mt aumberO {Vector edExcept ions ; . 

( return. aSrceptioas ) : } 

Universe • universe' ; 

{ returnf t!us->cpu!Jiu7erse ) ; } 

Thread * currentThre ad( ) 

{ returnr th is- >cpuCurr entT bread ),- 
Thread * idlaThreadf ) 

{ return' th is- > cpuIdleThr e ad } } 

Space * heap5pace( ) 

' returns th is- > cpuHeapS pace j, } 
Exception * exception* mt vector 

i return, th i s- > cpuTh lag s [ vector ] i 
Store * cr ivateStore * ) 

{ return, th is- >cpuPr ivateStore 
Store * globalstore ) 

{ return; th is- >cpudlob al Store ) . I 
Scheduler * scheduler (j 

{ returnt th is- > cpuSchedul er ); } 

Thread * thre adToDel e te ( ) 

{ return, th is- > cpuThr e adT oDel et e ) 


set-value functions. 


void setUniversef Universe * U ) [ 

th is- >cpuUniverse = U; 


void setCur r entThread ( Thread * aThread ; 

this->cpuCurrentThread = aThread, 


void setIdleThread( Thread * aThread ) { 

th is- > cpuIdleThread - aThread,- 
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« sVCs.h; definition of SVC numbers -he Kernel recc^nues. 

* This file should go away when we add cross domain object calls 

* SHeider : SVCs.h v 11.1 37/05/24 22:28:09 russo Exp $ 

* S Locker : $ 

V 

/• 

* Modification History: 

* 5 Log • SVC s . h, v $ 

* Revision'll 1 97/05/24 22:23:09 russo 

* changed order of defines’ 

* Revision 11 0 97/05/2L 15:50:02 russo 

* ConsoLe input and private stores. 

* Revision 10.0 37/04/2 2 07:30:42 russo 

* New Spaces. Universes and CP'J objects work, Finally* 

* Revision 93 37/04/94 1505: 53 russo 

* Multiple threads and timer interrupts. 

* Revision 3.0 37/03/29 15:29:53 russo 

» _nev and _delete added for memory management Also class interrupts work. 

* Revision 7.0 37/03/25 1245. 16 russo 

* Fault handler hierarchy works, so does iaterproces sor vectored interrupts. 

* Revision 1.1 87/03/17 1407:50 russo 

* Initial revision 


ORIGINAL' PAGE IS 
OF POOR quality: 


! I ifr.de f SVC s_h 
define 3VCs_h 

ildefine PRINTF SVC 0 

| (define X I L LTH XEAD_svc 1 
((define XI LIT ASK SVC 2 
Ildefine 3TART?HBEAD_SVC 3 

I 

ildefine LAST_3VC 3 

lendif SVC s_h 
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* Faults andler . h - rault handler class definition. 

* SHeader: FaultKandler h , v ll.l 87/05/24 05:07:57 russo Exp 5 

* SLocker: $ 

*/ 

* Modification history: 

* SLog: FaultHandler . h , v $ 

* Revision ll.l 97/05/24 05:9757 russo 

* adjusting tto reflect new ideas about fault handlers. 

* Revision 11.0 87/05/21 15:49:38 russo 

* Console input and private stores. 

* Revision 10.0 87/04/22 07:33:23 russo 

* New Spaces, Universes and CPU objects work, FiaalLyr 

* Revision 9.0 97/04/04 15:05:21 russo 

% Multiple threads and timer interrupts. 

* Revision 3.0 87/03/29 15:29:49 russo 

* _new and _delete added for memory management. Also, class interrupts work. 

* Revision 7.0 37/03/25 12:46:09 russo 

% Fault handler hierarchy works, so does interprocessor vectored interrupts. 

* Revision 61 97/03/22 12:21:48 russo 

* initial revision 


, llfadef FaultHandler_h 
Idefme FaultHandler h 


! I include 'Object. h“ 
it include ’ Store, h." 


// parent class 


(class Space.- // include Space h would cause a Circular definition. 

i/* 

I * Fault Handler Parent Class. This just defines what the rest of the kernel 
w thinks a fault handler interface looks like. Individual derived types 

* can specify all kinds of way to handle the actual faults, as long as 

* they meet the interface described here The parent class implementation 

* of firFauit currently Halts the processor. 

V 

class FaultHandler ; public Object { 
public: 

virtual void fixFaulti Space *• space, void *• address ) ; 


*• Fault handler to manage allocat ion/deallocat ion from a store. 
* This is about as simple as they get. 

*/ 

class StoreManager public FaultHandler { 


' h/F iui tHardler . a Pig? 2 


; May 2 5 ; 5 : j 5 19 3 7 


:roi?c:?d' 

j'tar? * st or ?3e mgMan aged . 

i publ 1 C : 

Stoce«atug?r / Store * ); 

’StorsMiaager; : 

void fixFauiti Space * space, void * address 


/* 

« Fault handler subclass to fill on demand a faulting page 
; « The only thing actually implemented here is the code i o allocate a 
j * page from tha store and then map it into the current tasks address 
j * space at the faulting address. 


! class Demand? illFault Handler : public FaultHandlex { 

j p U b 1 1 C : 

void aliocateAhdMap( Space * space, void * address ) 


■ A suDciass of demand fill fault 
j * constructor chat defines how to 
* allocated and mapped in. 


j I include " Filler. h 


handler that takes 
fill the faulting ; 


age 


Her m the 
once in is 


[class Demand? illerCI a s sFaultH andler public Dema ndF illFauitHa ndier { 

Filler * f ill if , 

; publ ic : 

i DemandFillerCl assFaultHandler ( Filler • filler 

| 'DemandFillerClassFaultHandler ( ) ; 

■ void f ixF ault < Space * space, void * address 


! 


I 


* A subclass of DemandFiilFaultHandler that fills a faulting page with 

* zeros once it is allocated and mapped in. 

V 

class DeraandZeroF aultKandler ; public DemandFiilFaultHandler { 
publ 1 C : 

DemandZeroFaultHandier ( ) 

~ Demand! ;ro? aul tHandi er ( ),- 
void f lx F aul t < Space * space, 

\ : 

lendif FaultHandler h 


Void * address 
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File.h: the parent file class definition. 

SHeader: File.h,v 11.0 97/05/21 15:49-44 russo Exp $ 

SLocker: $ 


Modification History: 

SLog : File . h . v S 

Revision 11.0 37/05/21 15:49:44 russo 
Console input and private stores. 

Revision 10.0 37/04/22 07:33:26 russo 

Mew Spaces, Universes and CPU objects work. Finally' 

Revision 90 87/04/04 15:05:25 russo 

Multiple threads and timer interrupts. 

Revision 30 87/03/29 15:29:52 russo 

_new and _delet# added for memory management. Also, class interrupts work. 

Revision 7.0 87/03/25 15:15:11 russo 

brought revision number up to date 

Revision 1.1 87/03/25 15:15:01 russo 

Initial revision 

/ 


llfndef File_h 
i (define File_h 

(include 'Obgect.h" // parent cliss 


void H a 1 1 ( ) ; 


jclass File : public 
j publ ic : 

j virtual mt 

! virtual mt 

virtual mt 


Cbgect ( 

readSecordsi long startSecord, void * 
vr iteRecords< long startSecord, void 
getRecordS ize( ) / 


buffer, lot count ) 
buffer, int count } 


class MemoryF lie : public File [ 
char * location; 
int length; 

publ 1 C : 

MemoryFile( char * location, mt length ) . 

’MemoryFilei > ; 

int readRecords/ long startSecord. void • buffer, int count ),- 
int wr iteRecords( long startSecord, void • buffer, int count ) ,• 

) ; 


lendif File_h 



$Header Filler h,v 11 5 97^05/21 15 4 9 -47 russo ixc 5 
$ Locker : $ 


* Modification history: 

* $Log: Filler. h,v S 

* Revision 11.0 37/05/21 13.4947 rusjo 

* Console input and private stores 

* Revision 10.0 37/04/22 07:3329 russo 

* New Spaces, Universes and CPCT objects work. Finally ' 

* Revision 9-0 97/04/04 15:05:27 russo 

* Multiple threads and timer interrupts. 

* Revision 30 87/03/29 15:29:54 russo 

* _nev and _delece added far memory management. Also ol»ss interrupts work. 

* Revision 7.1 37/03/29 15 4435 russo 

* Filler class definition move here from Faults andlerh 

•/ 


i i fr.de f ? ill er_h 
Idefine Filler_h 

I include "Object. h” // parent class 


* Filler parent class for use by the OemandF il 1 erci assF aultS andler class. 

* Maybe the default fillPage' can do something aue like fill it with 

* zeros'* Salting seems a little drastic. 

«/ 

extern vo id Haiti ) ; 


class Filler : public Object [ 

publ ic 

virtual void fillPage< void * fault ingAddress ) { 

Printf ( "Filler: : f illP age ' Ax ) c a 11 ed ! ’ \ n" 
fault ingAddress ), 

Halt ( ) s 


I 


/♦ 

* Filler to fill a COFF section from a COFF file. 

V ' 

I include ’ F lie . h'* 

class COFFSect ion? iller public Filler { 

File • file; 

void • sect ionStart ; 

int sect loaLength // m bytes, 

long f ileLocation,- // the location into the COFF file to load fr< 
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publ ic 

COFFSect ionFill«r( File * file, void * start, int length, 
long location ) ; 

“COFFSect i anFiller ( ) / 

void fillPigei void • address ), 


lendif Filler_h 
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* Lock.h - low - level spin lock 

* S Header; Lock.h v 11.7 37/05/26 0 5:53:59 russo Exp $ 


* / 

Slacker: S 




Modification history: 




SLog : Lock b , v S 



* 

Revision 11 .7 37/05/26 

0 5: 53 : 5 9 

russo 


Made heldByAnothsr an out 

1 me • 



Revision 11 5 37/05/25 

19:31:57 

g ohnston 


Added heldByAaother ; ■ . 



1 * 

Revision 115 37/05/25 

13:38:09 

3 ohnston 


Added heldByr ) . 



j . 

Revision 11.4 3 7 /05/25 

06:52:1' 

g ohnston 

1 • 

Made heldByMe a non- inline 


1 I 

Revision 11.3 37/05/25 

06:22-59 

■johnston 

j * 

ifcsdg fl skgdflg 



; I 

Revision LI 2 87/05/25 

06:17:13 

gohnston 

j * 

Fixed declaration of ThisCPU. ; 


* 

Revision 11.1 87/0S/25 

05:57:58 

gohnston 

j * 

Added heldByMe. ) . 



1 * 

Revision 11 0 37/05/21 

15:49: 53 

russo 

! ; 

Console input and private stores. 



Rev is ic n 10 3 37/05/13 

20:57:13 

g onnston 


* Removed Lock free ( ) because it’s confusing. 


; * Revision 10.0 87/04/22 07:33.10 russo 

j * New Soaces, Universes and CPU objects work. Finally' j 

! * t 

; * Revision 9.0 37/04/04 15:05:29 russo | 

j * Multiple threads and timer interrupts. 

i * | 

i • Revision 3.7 87/04/03 06:35:32 russo 

| * CFRONT is screwed up j 

• Revision 3.5 37/04/02 22:02:3 5 ]Ol»nstoa 

j * Added inline definitions of held') and free* ) . j 

I * Revision 1.1 37/04/02 15:19:59 gohnston j 

| • Initial revision ' 

i V 1 

| | if ndef Lock_h ! 

j (define Lcck_h j 

jextern unsigned ThisCPUf); 
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(class Lock { 

| unsigned char state; 

i unsigned mterruptState,- 

unsigned holdingCPU 

publ 1C : 

I LOC*(j; 


// The actual Lock. 

// Interrupts on flag. 

// CPUID holding lock (if locked). 


void acquirei ) .* 
to id releaser ) ; 


1 

f * The use of any of these will probably causes r ac es ... beware . 

*/ 

int he!d( ) 

{ return ( this-> state 6 Oxl ); } 

unsigned heldByt ) 

{ return ( this->holdingCPtJ ) ,• } 

int heldByMe ( ) 

{ return ( th is- > hold mgCPfJ == ThisC?tJ(> ) ,■ J 
i int heldByAiiother( ) ; 

u* 


j | end if 


Lock_h 


I 

1 


I 
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;/• 

* T ;ae: . h - ? e r - C ? T J timer. 

* SHeader: Timer. h,v 11.0 37/05/21 15:50:22 russs Exp 3 

* SLocfcer: S 

*/ 

/* 

* Modification history; 

* SLog • T imer . h , V 5 

* Revision 11.0 37/05/21 15.50:22 russo 

* Console input and private stores. 

* Revision 10.0 37/94/22 0733:55 russo 

* Mew Spaces Universes and CPU objects work • Finally’ 

* Revision 9.0 37/04/04 15-.06-.08 russo 

* Multiple threads and timer interrupts 

* Revision 11 37 04/03 09:19:25 Johnston 

* Initial revision 
*/ 

I if ndef T inter _h 
> Idef me T imer_h 

| I include 'Object. h” 

ioiass Timer : public Object { 

putal ic : 

T imer' ) ,■ 

~T uner{ ) ; 

void start{ unsigned milliseconds, unsigned vector } 

] vo id stop ( > . 

i nt idle-); 


ORIGINAL’ PAGE IS 
OF POOR QUALITY! 


S 


i 

i 

! 

| 


I 



j I end if Timer_h i 

! i 

i 




Itifndef Uaiversa_h 
||defme Uaiverse^h 


* For now the Universe has a fixed amount of spaces that can be mapped m 

* to fixed places. This will be extended when I have time. 

*/ 

class Universe : public Object { 

PTE * f irstLevelPageTable; 

Space * kernelSpace ,• 

Space * userSpics; 

publ ic : 

Umverset Universe * where. PTE * pageTable ); 

~Umverse< ) ; 

void addSpace( Space * aS pace ): 

Space * spaceCont a ining ( void * address ) ,• 
void loadContext For ( Thread * newThread ) ; 


lendif Universe_h 
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Space. h: ipi:s class description. 

5 Header Space . h v 11.' 37/05/26 21:41:33 russo Exp S 
SLocker; S ^ 

Modification history: r- 

5 Lcg : Space h , v $ 


Rev is ion 11.7 

37/0 5/26 

•2 1 

: 41 : 

33 



- 



Revision 11.5 

37/05/26 

05 

•04 

: 50 


changed name of alloc atsPo interT able argument. 

Revision 11 5 37/05/25 17:36:25 russo 
sorry, not implemented. 

Revision 11.3 87/05/24 23:13:38 russo 
more vork on new allocation stuff 

R-vision 11.2 37/05/24 16:46:27 russo 

redid allocation stuff and other private methods. 

Revision 11 1 37/05/24 05:13:57 russo 

added new allocate method definition 

Revision 11.0 37/05/21 1550:07 russo 

Console input and private stores. 

Revision 10.10 37/05/14 19:57:05 russo 

added isln' ; method for use to check vheather an address is managed by 
a space 

Revision 10.3 37/04/22 16 56:14 russo 

fixed XernelSpace constructor args. 

Revision 10 2 37/04/22 16 13: 38 russo 

switch constructor to take base and length rather than start and end 

Revision 101 37/04/22 09:36:41 russo 

renaming from MewSpace 

Revision 10 0 37/04/22 07:33:35 rusi 

Mew Spaces, Universes and CPU objects Finally* 

Revision 95 37/04/14 20:58:46 russo 

created KernelSpace subclass 

Revision 9.1 97/04/13 04:50:24 russo 

initial revision, (actually a rewrite of the old stuff) 


lifndef Space_h 
♦define Space_h 


Kay 26 21:42 1987 . ,/h/Space h Page 2 


I include 
I l nc 1 ude 
I include 
t include 
I include 


“Object. h" // parent class 

11 Lock . h" 

•vm. h" 

■ store . h‘* 

“ FaultHandler . h“ 


/* 

* Space - base class. 

1 V 


enum alloc at ionType { prefetch, faultln } 
class Space : public Object { 
friend class Universe,- 


protected : 

Lock lock; 

Store * store; 
void • baseAddress. 
int Length; 

unsigned vTopPage; 
struct { 

PTE f irstLevelPTE,- 
PTE • secOndLevelPTE ; 

} tablet 2 56 ] 

FaultHandler • f aultHandler [ MAXHAWDLERS ]. 

virtual void getPouterTables( unsigned lowpage, unsigned highPage 

virtual void buiidMapp ings ( unsigned mt start, unsigned mt count, 
FaultHandler * handler 


virtual PTE * allocatePomterTablef mt page i .- 

virtual mt convertFaultHandlerToI ndexi FaultHandler * handler j ,- 

publ ic • 

Space' Store * store, void * baseAddress, mt length ; 

"Space i ) ; 


virtual void * startAddress ( ) 
virtual void * endAddress ( ) ,- 
virtual int isla< void • vaddr ),- 
virtual mt isValidt void • vaddr ) „■ 

virtual void • allocate( unsigned mt count, FaultHandler * handler 
allocat ionType type )/ 

virtual void • aliocate( void • base, unsigned mt count, 

FaultHandler * handler, allocat ionType type 

virtual void map( void • page, void * frame ). 

FaultHandler * handler ( void • vaddr ; , 

/* these are dead */ 

virtual void • aliocate< unsigned int pageCount i 




i 


I 
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• Except ion. b; event cl*** description. 

* SHeader : Exception. h, v ii.O 87/05/21 15:49:35 rusaa Exp S 

* SLocker: 5 

*/ 

/* 

‘ Modification History; 

* SLog: Except ion . h , v $ 

• Revision 11. 0 87/05/21 15:49:35 rutso 

* Console input and private stores. 

• Revision 10.13 87/05/01 11:02:08 russo 

• renamed from Event. b 

V 

lifadef Exception^ 
j tdef ine Exception~h 

'♦include "Object. h" // parent class 

; I include "Thread. h" 
j I i nc 1 ude "Fr ame . h " 

•typedef void ■ • SandlerFunct ion )( struct Frame • frame ) 

class Exception : public Object { 
protected : 
publ ic : 

virtual void poat( struct Frame * frame ) • 


class SystemExcept ion : public Exception { 
protected : 

HaadlerFunct ion handler.- 

public : 

sy stemExcept ion{ HandlerFunct ion tbeHandler 

~Systea£xcept ion( ) 

void post( struct Frame • frame 


class InterruptExcept ion ; public Exception { 
| protected : 

Thread • a waiter,- 

publ ic : 

rnterruptExcept ion( ) ,- 
' 1 ater r up t Except ion ( ) ; 
void post( struct Frame • frame ),- 
void await ( ) ; 

}/ 

lendif Exception_h 


1 
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ipdceLuth Space - i s t class descriptian. A ScaceL .s an ordered 

list of Spaces It is specially coded md not a sub- 
class of some nor* abstract list class since it Lt very 
heavily used and high efficiency is desired 

SHeader: Sp ace! ist . h , v 11.1 37/0 5/2 3 20:13:26 russo Sxp $ 

SLocker: $ 


/ 


Modification History 

$Log: Space! ist . h , v $ 

Sevision 11.1 37/05/23 20 18 25 russo 

added enclosing ifdef 


Revision 11.0 37/05/21 15:50:10 russo 

Console mout and private stores. 


.Revision 10.2 5 7/04/2 3 2 1.1 5 2 2 russo 

Btade Object the parent class far pedantic 


Revision 10.1 37/04/23 2113:33 russo 

initial revision. 


Iifndef SpaceList_h 
Idefme Space!ist_h 

t include "Object. h‘ 

I include "Space. h" 

class space! istNode public Object { 
publ 1C : 

space! istNode * next, 
scaceL istNode * last . 

Space * data,- 

spaceL istNode' Space * s ) { 

next = 0 ; 
last ~ 0; 
dal a = 

} 


class SpacsList public Object { 
protected : 

space! istNode * head; 
space! istNode * tail; 
space! istNode * iterator; 
Lock lock; 


publ ic : 


Space! ist ( ) ; 

“Space! ist ( ) ; 

virtual void add( Space * space },* 
virtual int remove( Space • space 
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virtual int inquire! Space • space ) ; 


virtual void startlterat ion( ) ; 
virtual Space * iter ateNext ( ; ; 


fendif Space! ist_h 




i 


i 


i 
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,■ h/Frinie h Page l 


* Frame. h: description of the frame on the -aterrupt sties passed to 

* exception handlers. 

* SHeader Frame. h.v 11.0 37/05/21 15:45:50 russo Exp $ 

* 5 L o c k e r . $ 


I 

i 


* Modification History-. 

* SLog: Frsme.h.v S 

* Revision 11.0 87/05/21 15:43:50 

* Console input and private stores 


* Revision 10.7 37/04/3 0 14.55:15 

* added asp to frame. 

k 

« Revision 10.6 37/04/29 07. OS. 13 

* made all fields unsigned mtishoti 


« Revision 10.1 37/04/ 27 13: 33 03 

* initial revision. 

V 


russo 

russo 

russo 

russo 


, I if nde f "r ame_ 
: idef me * r a me 


Istruct Frame 


uns igned 

int 

*P 


uns igned 

int 

fP 


uns igned 

mt 

r7 


uns igned 

int 

r$ 


unsigned 

int 

r 5 


uns igned 

mt 

r4 


unsigned 

int 

r 3 


uns igned 

mt 

r 2 


unsigned 

int 

rl 


uns igned 

mt 

rO 


unsigned 

mt 

vectorNumber .- 

uns igned 

int 

pc 


unsigned 

| 

short 

mod.- /* th 

sb 

i unsigned 

short 

psr; 


so we dont save sb . 


lendif Frame_h 


ORIGINAL PAGE IS 
OF POOR QUALITY 
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i 

! 


t 
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/* 

* Vectors. h: vector numbers for processor traps/ interrupts . 

* SHeader: Vectors.h,v 11.0 87/05/21 15:50:35 russo Exp $ 

* SLocker. $ 

*/ 


* Modification History: 

* $Log: Vectors. h,v $ 

* Revision 11.0 97/05/21 15:50:35 russo 

* Console input and private stores. 

* Revision 10.5 87/05/13 13:47:59 johnston 

* Added definition of CPUCLASS_MAX for assertions, etc. 


* Revision 10.4 87/05/13 13:45:49 johnston 

* Changed CPUCLASS to be CPUCLASS_HALTED (0) and CPUCLASS_RUNNING (1>. 

• Did this so that a halted CPU won't get class interrupts that will 

• never get serviced. 


; • Revision 10.3 37/04/30 16:25:26 

j * Added CPOCLASS definition. 


{ * Revision 10.1 97/04/28 09:28:06 

| • initial revision 

i */ 

'lifndef Vectors_h 
i Idef ine Vectors_b 


• Trap 

Vec 

tor3 


(define 

MV I 

_V«ctor 

0 

(def me 

MM I 

“vector 

1 

(define 

ABT 

^Vector 

2 

(define 

FPU 

“vector 

3 

(define 

ILL 

“vector 

4 

(define 

SVC 

“vector 

5 

(define 

DVZ 

“vector 

8 

(define 

FLO 

“vector 

7 

(define 

3PT 

“Vector 

9 

Idef ine 

TRC 

“Vector 

9 

(define 

UNO 

Vector 

10 

(def ine 

RESERVED 11 

^Vector 

(define 

RESE RVED"~12 

^Vector 

(define 

RESERVED 13 

^Vector 

(define 

RESERVED 14 

“vector 

(define 

RESERVED 15 

J/ector 


11 
12 
13 
L 4 
15 


johnston 


russo 


/* Mon- Vectored I titer r up t */ 

/* Mon-Maskable interrupt */ 

/« Abort ;7M Error; Trap •/ 

/* Floating Point Exception Trap • / 

/• Illegal Instruction Trap */ 

/* Supervisor Cali Instruction Trap •/ 
/* Divide by Eero Trap •/ 

/* Trap on Flag V 
/* Breakpoint Trap */ 

/* Trace Trap */ 

/• Undefined Instruction Trap */ 

/ * Reserved */ 

/* Reserved */ 

/* Reserved */ 

/* Reserved */ 

/* Reserved */ 


/• 

* Interrupt Vectors 

*/ 

|de fine TIMESLICE_Vector 
(define CONSOLE^Vector 


16 /* Time Slice Counter Interrupt */ 

L7 /* Console Input Interrupt */ 


i 


! 


i 
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* det me 

UNUSED 12 Vector 

1 3 


Idef me 

UNUSED 13 Vector 

19 


1 define 

UNUSED 2 0 _Ve c tor 

20 


/* 




* Vector bus classes. 



1 define 

CP UC LASS HALTED 

0 

/* Halted CPU class */ 

1 de f ine 

CPtJCLASS RUNNING 

1 

/* Normal running CPU class 

Idef me 

CP UC LASS ^MAX 

1 

/* Highest class used */ 


'tend if '/sctors h 




i 


i 

I 

i 




i 

i 
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| * Scheduler.il: a dispatch queue description. 

i * 

; * SHeader: Scheduler . h , v 11.3 87/0 5/2 1 15:50:04 russo Exp $ 

I * S Locker : 5 

; v 

I * Modification History: 
j * SLog: Scheduler . h , v $ 

| * Revision 11.0 87/Q5/2L 15:50:04 russo 

j * Console input and private stores. 

| * Revision 10.2 87/05/12 17:44:04 russo 

I * added enclosing I lfdef 

j * Revision 10.1 87/05/04 17:21:20 russo 

I * initial revision 

! *' 

'lifadef Scheduler_h 
tdefine Schedulec_h 

! I include "Thread. h" 

I 

class Scheduler { 

protected : 

Lock lock, 
mt next In; 

! int nextOut, 

mt nisThre ads 
j Thread * * Queue, 

publ ic : 

Schedulers mt maxThreads 

“Schedulerf ) 

void add; Thread * ); 

[ Thread * removeNext( j ,■ 


lendif Scheduier^h 


. / h/?r ivaceMemory . h Page 
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* ? r 1 v a teMemor 7 h Inline functions and constants to ;aicuiite the of 

* of various things into the private virtual memory area 

* BootPhys leal • ; and BootV irtu ai , ) use these to cooperate on where tn 

* SHsader : Pr 1 vateMemory h , v 11.2 87/05/26 06 46:01 russo Sxd $ 

* SLocker: S 

*/ 

/* , : 

* Modification History: 

* Slog: Pr ivateMemory . h . v $ 

* Revision 11.2 37/05/24 06:46:01 russo 

* males atacx 2 pages large 

* Revision 11.1 37/05/23 20:17.42 russo 

* added surrounding ifdef- 

* 

* Revision 11.0 87/35/21 15:49:59 russo 

* Console input and private stores 

* Revision 10.2 37/05/14 17 47.23 russo 

* ldded an extra const for the second page table page 

* Revision 10.1 37/05/14 17:26:41 russo 

* initial revision 


tifndef Pr 1 vateMemory_h 
♦ define P r ivateKemory^h 

♦include "Store. h" 

! * include 'CPU .a 1 ’ 
j I include ’YM.fc'* 

| ♦ include ■ Universe . tr* 
♦include Thread h" 



Map of the per-cpu private memory area. 


+■ * <<-- End of private 7M 

As many pages as needed to store 
the rest of the private store 
state information. The of memory 
up to the end of the cer-CPU 
private area is available. 

If more things need added to this 
area, add them below the private 
store and bump it up higher 

+ <<-- Base + 10 pages. 

! The beginning of the private 
[ store state information 


go 


ORIGINAL PAGE IS 
OF POOR QUALITY 
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<<-- Base + 9 pages . 


<<-- Base +■ 8 pages. 


<< — Base * 7 pages. 


< < — Base + 5 pages 


<<-- Base *■ 5 pages. 


<< — Base + 4 pages. 


<<-- Base *- 3 pages. 


<< — Base + 2 pages 


<< — Base + 1 page. 


<<-- Base of private VM area 

V 

/* 

* The CPU object. 

V 

const int mePage = 0 ,• 

static inline CP 0 *• meLocation( char *• base ) 

I 

return! (CPU *) ( base +■ 0 ) ) ,• 


Intentionally unmapped to catch 
stack underflow 


The Oerm Threads stack page 2 


The Germ Threads stack page l 


Intentionally unmapped to catch 
stack overflow 


The Germ Thread objects state 
mformit ion 


Second page of the CPUs 
page table 


j First page of the CPUs 
j page table 


Pointer table to point to all 
of the stuff here 


CPU and Universe objects state 
information 
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* The Jmverse ob] ■ect 


[const int universePage - 3,- 

i static inline 'Jmv-rse * uaiverseLoc at ion( char 


return: (Universe 


sizeoC : CPU 


j * The pointer table to access private things vith. 
Iconst mt pr iv i t e?o mterTabl eP age = L ; 

• static mime PTE * pr lvatePo interT ableLoc at ion <. char 

, { 

return r (PTE *1 ( base +■ PAGESIZE - 

i > 


ORIGINAL PAGE IS 
OF POOR QUALITY 


* This processors page table. 

const int pageT able? age = 2 ; 
j const mt secondP age? aoieP age = 3 ,• 
static inline PTE * pageTableLoc at ion( char 


base ) 

base * pageT abi eP age * PAGESI Z E 


l * The initial thread. 


“const int germThre adPage = 4, 

istitic inline CermThread * germThr a idLoc at ion ( char 


return: (GermThread *) 


base ) 

base * germThr e adP age* PAGES I ZE ) 


The stacic Cor the initial thread 


j const mt stacfcPage - 6 ,• 

I static mime char * st acicLoc a t ion t char * base ) 


return; ( char 


base * st acteP ag e* PAGES I ZE ] ); 


:/* 


hate information 


1 * The private Store 

t */ 

,const mt pr ivateStoreP age = 9. 

■static mime Store * pr lvateStoreLocat ion ( char 


base 

return' (Store *) ( base * privateStor eP age * PAGES I ZE ) ) .- 



I 


ORIGINAL PAGE IS 
OF POOR QUALITY 
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KecrvelKa l.i( ! : 

This routine sets up things that need initialized before 
the processors are all let loose 

SHeader: Ker nelMa la . c ,v 11.4 37/05/25 55:17:33 russo Exp $ 

S Locker $ 

Revision History: 

SLog KernelMain. c , v $ 

Revision 11.4 87/05/25 05:17.33 russo 

only do dselective debug printing 

Revision 11.3 87/05/24 06:13 : 55 russo 

took out VERY annoying debug 

Revision 11.1 37/05/21 15:51:10 russo 

switch ways of deleteing threads 

Revision 11 0 37/05/21 15 54 43 russo 

Console input and private stores 

Revision 10.32 37/05/17 14:04:26 russo 

keep deleteQueue as a per-CPD member 

Revision 10.24 97/05/13 20:29:56 russo 

Split out the time slice thread code and did soma other rearrangements. 

Revision 10.20 97/05/13 06:07:23 johnston 

Added creation of console thread. 

Revision 10 17 87/05/12 17:35:26 russo 

restructured to be run by the germs initial thread. Also sets us up 
to use the new scheduler;) member of the CPU class. 

Revision 10.0 37/04/22 07:38 16 russo 

New Spaces, Universes and CPU objects work. Finally* 

Revision 9.0 97/04/04 15 12:29 russo 

Multiple threads and timer interrupts. 

Revision 3.0 37/03/29 15:33:45 russo 

_new and _deiete added for memory management. Also, class interrupts work. 
Revision 7.0 37/03/25 12:49-34 russo 

Fault handler hierarchy works, so does interprocessor vectored interrupts. 

Revision 1.1 87/02/23 23:34:11 russo 

Initial revision 


nclude "Assert.h" 
nclude 'Debug h" 
nclude " md_tuneable . h" 
nclude "Store. h" 


original page: is 
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» include "Flle.h" 

I include ’ FaultHandler . h“ 
I include "Space. h" 
linclude "Task.h" 

I include "Scheduler . h" 
linclude ' Except ion . h" 

I include "Lock h" 
linclude "tlaivarse.il" 
linclude CptJ.h" 
linclude "Vectors. h" 


* Global Exception handlers. 


extern void ABTT r ap ( struct Frame % ); 
SysteroExcept ion VMExcept ion{ ABTTrap ) ; 


extern void SVCTrap( struct Frame * ) ; 
SystemXxcept ion SVCExcept ion( SVCTrap J ,■ 


InterruptExcept ion ConsolelntarruptExcept ion,- 


* Miscellaneous variables. 


static iat setupDone = 0 , 
static Lock setupLocfc; 
static Scheduler * runQ; 


// run queue for this kernel 


static int ConsoleThreadBuilt = 0; 
static Lock ConsoleThreadLock; 


typed* f void (• KPFV)(),- 


void 

KernelMain( unsigned iat ID ) 


* Initial sanity Assertions and Debugging. 

V 

Asseitf Me != 0 ) ; 

Debug( "Processor %x has noined the kernel (ID=%x)\n", Me->id(), ID 
Assert ( ID == Me->id() ),- 
Assert ( Me- >universe( ) '= 0 ) „• 

Assert; Me- >heapSpace( ) '= 0 ); 

Assert; Me-> currentThread; ) ’=0 ); 

Assert; Me- > ldleThread; ) *= 0 ) , 

Assert; Me- >currentThread; ) == Me- > idleThzead ( } ) ; 

Assert; Me- > scheduler () =» 0 ); 

Assert; Me- > threadToDelete; ) == 0 ); 

Assert; Me- > pr ivateS tore ( ) := 0 ),- 

Assert; Me- >globalStore( ) 0 j ; 


extern int InterruptsDisabled; ) , 
Assert; InterruptsDisabled;) ),- 
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» Install the glcba! (common amoung all C P 7 s executing this kernel) 

* exception handlers. The ABT handler is especially important since 

* it will allow page faults to occur and be handled. 

* / 

Debug; 'KernelMam: Installing global except ions\n ' ); 

Me- > setExcept ion( ABTJ/ector, iVMExceptlon ); 

Me- > setEscept ioa( SVC_Vector. iSVCExcept ion i 

Me- > setExcept ion( CONSOLE_Vector , ^Console I nt sr rupt E xc ep t ion ),- 


/* 

* install the local (private) exception handlers. 

* Suild a stack for the clock thread, then construct and dispatch it. 

* It will build, install and await the time slice exception. 

*/ 

Debug ( "KernelMam: Building kernel clock thread\n" ) .- 
extern void SwitchTof Thread * ) .- 

extern void * XernelThreadSt ackAlloc atcr ( mt )■ 
extern void Ticker < mt \: 

mt * stack - (in t •> KernelThreadSt ackAlloc ator 1 i. 

Assert stack • = 3 > ; 

Thread * clock = new I nter ruptThr e ad ( fPFV) Ticker, stack, 3. 0 0 ) ,■ 
Debug; "KernelMam: clock thread *.x ( st ack= >sx ) \ n “ , clock, stack ).- 
Assert, clock * = 0 ) ,- 

SwitchTo( clock ) ; 

Debug< ’'KernelMam: clock dispatch returned\n" ); 

Assert) Me->id() == ID ); 

/* 

* 3uild a stack for the console thread then construct and dispatch it. 

* / 

ConsoleThreadLock . acquire) ) 
iff ' Consol eThreadSu lit ) { 

/* 

* If were the one going to build it. indicate so and let 

* everyone else procesde since there is no reason for then 

* to wait for us. 

*/ 

ConsoleThreadBuilt = 1; 

ConsoleThreadLock . release ( 

Debug ( "KernelMam: Building console thread\n" ) .■ 

stack = ; int *) KernelThr e ads t ackAi 1 oc a t or L w 
Assert) stack ’= 0 ); 


extern void ConsoleThreadEntry ( mt ; , 

Thread • console = new I nt errup tTh re ad ( 

)PFV» ConsoleThreadEntry, stack, 0, 0, 0 » ,• 

Debug( KernelMa in : console thread u <; stack= )\n“ , 

stack ) ; 

Assert) console ’=0 j; 
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SvitchTo( console ) ; 

Debug! "KernelMam: console dispatch returned\n" ),- 

Assert) Me->id() == ID ) ,• 

) 

else [ 

ConsoleThreadLock . release ( ) . 

) 

/* 

• Check to see if the setup portion of the kernel has been done by 

• someone else and if not do it. Otherwise wait for them to finish 

• then continue on. 

V 

setupLock . acquire) ) ; 
if( ! setupDone ) { 

Debug( "KernelMan; Doing kernel setup\n" ),- 
setupDone = 1/ 


* Initialize the run queue scheduler. 

V 

runQ = new Scheduler ( LOO // hold 100 threads. DEFINE THIS 1 ! * 
Debug ( "KernelMam: runQ = *x\n " , runQ ),- 
Assert) runQ * = 0 ); 

/* 

* Create the initial Space. 

V 

CPOPrintf{ "KernelMa in: Creating mit space from *x\n'‘ , 

Me- >globalS tor e ( ) i ; 

Space * mit i alSpace = new Space) Me- > globalStore r < . 

(void •) TASKLOWADDR. TASKHIGHADDK - TASKLOWADDR ) 

Debug ( "KernelMam: mitialSpace = vz\n' mitialspace , 

Assert) mitialspace 9 j 

/* 

* Create the File to load the space from. 

•/ 

extern char I nit lalCode ( ] 
extern int I n it lalCodeS i ze 

Debug( "KernelMam: Creating the initial space' s COFF file\a*’ ), 
File • initialFile = new MemoryFilef InitialCode, 
iaitialCodeS ize ) .■ 

Debug ( "KernelMam: initialFile = Vx\n" . initialFile ), 

Assert) initialFile '=0 ) ,■ 

/* 

* Initialize the space from the COFF image of the initial Task 
» in the file created above. 

V 

extern KPFV SetupSpaceFromCOFFImage( Space *, File * ) ,■ 


Debug( "KernelMam: initializing space from the COFF image\n" ) . 



I 


I 
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KPFV eatryPomt = Setups? aceFrsmCOEKrniag* mtulSpate, 

> : 

Debug! 11 Ker aeiM ua • entryPoint = •sxNn" entryPoint ! 
Assertt iatr yPo vat •- 0 ) ,- 

/* 

* Great* a taste int the rn.rti.al space. 

•/ 

Debug! " Ser nelMa r n : Creating initial task'tx. %x;\a" . 

mitialSpace entryPoint 
Taste * imtialTask = new Taste! in it lalSpace . 

ZPFVi entryPoint ),- 

Debug! "KernelMain: imtialTask = *iX\n’, inrtialTastc ) 

A ii«rt( mitialTastc ‘=0 ); 


* 2 nqueue the tastes initial thread on the scheduler we 

* made above 


Deoug< "KernelMain- adding initialThr ead >x to runQ\n", 
inrt ialTasi-> mit iaiThread( ; ), 

Assert! in it i alT ask- > m i t laiThread ( ; !* 0 ),- 

runQ- > add ( initialTask-> initialThread; ) 


set up Lock . release ! j 


original page is 

OF POOR QUALITY 


/* 

* Now 

* and 

* CPTJ 
*/ 

extern 


that all the initial interrupt threads have been dispatched 
started, turn on interrupts, set the scheduler member of the 
object to the kernels scheduler and begin round robin scheduling. 

void Eaablel nt errupt s { ) ,- 


Debug! "KernelMain: enabling mt errupts\n" 

Enabletacerrupts. ),■ 

Me- > setScheduler i. runQ ); 

/* 

• Keep looping removing things from the scheduler and dispatching 

• them. 

V 

Debugt 'KernelMain: entering idle loop\n" ).- 
vhilii 1 ) l 

extern int InterruptsEnabled( } .• 

Assert! InterruptsEnabledf ) ) ,- 

Thread • aThread; 

Assert ! Me- > schedul er ( ) • * 0 >/ 

Assert! Me- > thceadToDelete! ) =» 1 v , : 

while! ( aThread = Me- > scheduler !)-> removeNext ( ) ) == 0 ). 

CP f JPr intf ( "Idle!): aThread = *x\n" , aThread ),- 

SwitchTof aThread ) ,- 

CPUPr intf ( ■•Idle!): RETURNED, Checking for Threads to delete\n" ) ,- 

Assert! Me->id() = x ID ) 

Assert! Me- > currentThread ( ) -= Me-> idleThreidf ) > ; 
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• Check to see if there is a Thread to delete. 

V 

if! ( aThread = Me-> threadToDelete( ) ) 0 ) { 

CPOPrmtf! "KernelMain: deleting aThread ), 

delete aThread,- 

Me- > setThreadToDelete! 0 ),* 


/• 

* The rest of this file is only here for testing. 

* it belongs :in. and will be moved to, somewhere else in the kernel when 

* it is completed. 

•/ 

/ * 

* Suild a stack for a kernel {interrupt) thread. The stack is allocated 

* virtually in the processors current heap space and physically from 

* Me- >globalStore( ) .The pointer returned is to the start (bottom) of the stack. 

* This is actually the high end of the memory region allocated. 

* FIX THIS TO ALLOCATE "SANDWICH" VIRTUAL PACES TO CATCH OVER/UNDERFLOW' 

* AND TO USE THE PROPER STORE 
*/ 

vo id * 

Ker aelThre adStackAl locator ( int numberO f P ages ) 

Debug! 1 CernelStacfcAllocator ( *d )\n" , numberO fP ages 
Assert! Me- >giobalStore ( ) * = 0 

Assert! Me- > heapSpacei ) •= 0 ); 

char • vstack = < char •) Me- > heapSpace ! )->allocat«( numberOfP ages ) : 
void * pstack = Me- >globalStore( ) ->allocate( numberO fP ages W 
Assert! vstack 1 = 0 ) ,- 
.Assert! pstack '= 0 

Me- >heapSpac«< )->map< vstack, pstack, numberO fp ages ) ; 
return! vstack * ( numberOfPages< <PAGESHIFT) ) ,- 

} 
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Store c - physical memory allocation. 

SHeader: Store. c,v 11. 0 3 7 /05/21 15:57:13 russo Exp $ 

ORIGINAL PAGE IS 
OF POOR QUALITY: 

ruiio 

j * Revision 10-5 97/05/15 05: 55': 58 johnston 

I • Fixed calculation of Store state page count in constructor. 

j • Revision 10.3 87/05/14 21:56:09 Johnston 

' • Hopeful/ fixed some stuff. Changed the interface, too. 

i • Gotta hunt around and find all the places I broke it. 

I * 

! * Revision 10 . 0 37/04/22 37:43:34 rusao 

• New Spaces, Universes and CPU objects work. Finally' 

| * Revision 9.3 37/04/04 15:17:26 russo 

• Multipie threads and timer interrupts. 

( • Revision 8.0 87/03/29 L5-.37-.44 russo 

• _nev and ^delete added for memory management. Also, class interrupts work 

• Revision 7.0 87/03/25 12:52:55 russo 

• Fault handler hierarchy works, so does interprocessor vectored interrupts. 

• Revision 1.1 87/02/23 18:20:25 russo 

• Initial revision 

i V/ 

' I include 
[ f include 
I include 
I include 
I include 
! I include 

Store :: Store { uns igaed basePage, unsigned pageCount, unsigned * st ateBaseP age ) 

( 

/* 

• Debugging and entry assertions. 

*/ 

Debug { " Store :: Store( %x, AX . ax )\n" , basePage pageCaunt, st ateBaseP age ),- 

J Asserti st ateBasePage ) , 

j Debugt " Store :: Store : * stateBaseP age: Ax\n " , * st ateB aseP age 1; 

I Assert ( pageCaunt ’ = 0 ) 

Assert! this == 0 ) 


"rad_const ants . h" 
"Assert . h" 

"Debug . h." 

" lock . h" 

" store . h" 

"VM. h" 


Revision History: 

SLog : ' STIOre-. c , v 5 

Revision 11. 0 87/05/21 15: 57 : L8 

Console input and private stores. 


* Figure out our sires, etc . 

•/ 
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unsigned pagesPerSetEntry = 8 • sizeof( unsigned 

unsigned sstEntryCount = ( pageCount + { pagesPerSatEntry - 1 ) ) / 

pagesPerSetEntry ,- 

Assert: ( settntryCount • pagesPerSetEntry ) >= pageCount ); 

unsigned st ateByteCount = { $iteof( Store ) - sizeoff unsigned ) ) * 

( setEntryCount • sizeof( unsigned ) ),- 

stateByt-Count = PageCeilingl st ateByteCount 

unsigned st ateP ageCount * addrToPage( (char •) st ateByteCount ),- 

• Allocate ourself 
•/ 

this a (Store *) ( * stateBasePage • PAGESIZE J .* 

* st ateBasePage += statePageCouat; 

Debug ( “store: : store: this: Ax < Ad pages>\n", this, stateP ageCount 


/ 

* In 
*/ 

th is- 
th is- 
th is- 
this- 
this- 
f or ( 

Debug 

Debug 


itialize the Store state information. 

>baseP»ge = basePage,- 

>highPage = basePage +■ pageCount, 

> freePageCount = pageCount,- 
> setEntryCount * setEntryCount; 

>pagesPer3etEntry = pagesPerSetEntry. 
unsigned i = 0,- i < this->setEntryCount . i++ » 

this- > set [ i ] =0/ 

( "Store: j Storet basePage-. *x, highPage-. Ax, freePageCount: Ax\n" 
th is- > baseP age , this->highPage, th is- > freePageCount ),- 
( " Store Store : setEntryCount : %x, pagesPerSetEntry; Ax\n" , 

th is- > setEntryCount , th is- >pagesPerSetEntry ) „• 


! Store : 
' { 


“Store ( ) 

extern void Halt() ; 

th is- >lock . acquire ( ) 
P an lcpr int f ( "Store; 
this = 0 1 
«*lt( )/ 


“Store : DESTRUCTOR CALLED on Ax\n" , this ); 


• Store ; : nextfree 

• Return the first free page after or including the specified page. 

• Return th is- >h ighP age if none were found. 


uns igned 

Store; : next Free ( unsigned page 


Assert( page >= this- >baseP age )/ 
Assertf page < th is- > h ighP age ),- 
while ( page < th is- > h ighP age ) { 

if ( ! this->marked( page ) ) 
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return p a .2 e 

? i 5 94 '*-' 

returri r n is- > h i?bP age ; ,• 


/* 

* Store : ; coot iguous ^ 

* Return the number of contiguous free pages starting at the specified page 

* LooK: for no more than the number of pages specified 
V 

uns igned 

Store :: contiguous ( unsigned page, unsigned pageCount ) 

{ 

Assert ( th is- > inStore ( page , 

for ( unsigned count = 0; count < pageCount count + f, page*-*- } [ 

if < page == this- >h ighP age ) 
return •• count } 
if i th is- >taariced ; page i •, 
return •' count ) ■ 

Assert? th is- > instore? page • j ,• 
return ( count ).- 

} 

char * 

Store :: allocate! unsigned pageCount ( 

/* 

• First fit search for number of contiguous pages requested. 

Debug k '' Store allocate: j : this = 'X\a" pageCount, this ); 

Assert( pageCount *= 0 
this->locfc acquire ( • 

uns igned ba seP age = this->basePage.- 

unsigned f r eeP ageCount = 0; 

while ( basePage < th is- > h ighP age j { 

basePage = th is- > nextFr ee ( basePage ); 

free? ageCount = th i s- > cont iguous ( basePage, pageCount ) 
if ' f reePageCount == pageCount i { 

Assert; th is- > instore •' basePage ) ); 

unsigned page = basePage, 

for ( unsigned i = 0; i < freeP ageCount ,• i++ , page+ + 

this->mark( page • ,• 

char * addr = (char •) pageToAddn basePage ) 

Assert; th is- > instore < addr i ) ; 
th is- >lock . release; ) 

Debug( "Store : . allocate : %x\n", addr 

return ( addr ) 

} 

basePage •*■= free? ageCount ,• 

} 

this->lock. release! ) .• 

CPUPnntf( "Store :: allocate : FAILED -'this = *xi\n'. this > ,• 
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return ( 0 ) ; 

\ 


vo id 
Store : 
{ 


deallocate( char • baseAddr, unsigned pageCount 

Debug; " Store :: deallocate ( %x , \d ) ■ this = U\n", 
baseAddr, pageCount, this j ,• 

Assert ( ; (unsigned) baseAddr * PAGESIZE ) « 0 

unsigned basePage = addrToPage( baseAddr ); 
Assert { th is- > inStor e ( baseP age j ); 

Assert( this-> instore ( ( basePage v pageCount , 
this- > lock . acquire ( ) ,* 
while ( pageCount — ) { 

this->unaark ( basePage ); 
basePage+-+; 

} 

this- > lock . release { ) 


1 


) ; 


vo id 

Stor e :: reserve ( unsigned basePage, unsigned pageCount ) 

{ 


Debug< "Store: reserve; %x, %d) : this = %x\n", 
basePage, pageCount, this ); 

Assert( this-> instore ( basePage ) ) ? 

Assert ( pageCount •= 0 ) .• 

Assert( this-> mStore( ( basePage + pageCount ) 
this- > lock . acquire! ) : 
while ( pageCount — ) { 

this- >mark( basePage ) ,- 
basePage++ ; 

} 

this->lock. release! ) 


1 ) ); 


I 
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I include "Debug. h" 

I include ‘Assert. h" 

I include "VM - h" 

I include • Store h" 
t include "Space h" 

I include " FaultHandler . h” 
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! lock, acquire ;. ' Th.s could be ir:-.Lb> i - 2 processors 

/ try to delete the space at the same time. 

/* 

* Destroy all of this spaces fault handlers. 

* WHAT IF THESE SHARED. I GUESS THEY MEED TO 3E REFERENCE COUNTED 

V 

for f int X = 0 - i < MAXHANDLERS .• i + - ) { 

FaultHandla'r * f = f aul tH andl er [ i ] 
if.( f ■= 3 ) 

delete t; 


/* 

* Return any second level page tables allocated. 

*/ 

for; 1 = 0 ; i < 256; !++■ ) { 

if( th is- > table • i ] secondLevelPTE *= 0 ) { 

//WHAT ABOUT THE VIRTUAL SPACE IT OCCUPIES 
CPUPr int f •: "'Space; THIS REALLY DOESNT WORX\ n " ) .- 

void * paddr = pageTcAddr' 
th is- ' t abl e [ i] . i ir stLevelPTE . pageN umber ;. 
ft his->store)-> deal locate, char * paddr. L t. 


/* 

* See if an address fails within the range managed by a space. 

V 

int 

Space ■ : is t n i void • addr } 

{ 

Debug' " 3p ace • : xsl n ( 4 x this = UNn' , addr. this i .• 

Assert < this ' = 0 ) ; 

lfi ( addr >= th is- > b aseAddxes s \ i i ( addr < = 

(void * ) < -char *) th is - > ba seAddr e ss +■ thi3->len3th - L ) > ) 

return! I i ; 
return! 0 ) - 

} 

/* 

* See if an address managed by a Space is currently resident in physical 

* memory ; valid! • 

V 
int 

Space : : lsVal id( void • addr ;■ 

{ 

/* 

* Entry assertions and debugging. 

V 

Debug! " S pace : : isVal id{ \x ): this = %x\n" , addr, this ),- 
Assert( this '=0 )/ 

Assert! this->isla( addr ) ); 

/• 


I 

I 

I 
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} 


* See if the 
•/ 

int lllndex = 
int 12Index = 


space mappings think the page is 

v (VA) addr ) . f irst Level I ndex( ) 

{ (VA ) addr ) . secondLevell ndex( ) ; 


valid. 


if! th is- > table [ 1 1 1 ndex ]. secondLevel PTE { 121 ndex ]. val id( } ) 

return! 1 ) ; 

else 

return! 0 ) ; 


/* 

* Return the first address managed by a Space 
*/ 

vo id * 

Space; : startAddressf ) 

{ 

return! th is- > basaAddress ); 

) 


/* 

* Return the last address managed by a Space. 

V 

void * 

Space: : ecdAddress! ) 

{ 

return! (void *) ( (char *) this- >baseAddress +■ this- >length - 1 ) ) .- 

} 


i 


/* 

* Allocate "count" random pages from the space of type "type" 

* Use 'handler" to manage them. 

*/ 

vo id * 

Space allocate! unsigned int count, FaultHandler * handler, 
allocationType type ) 

| Debug! " Space ;: allocate! counted handler : type : ad ): this=%x\n' 

count, handler, type, this 
if( count ss 0 ) 

return! 0 ) ; j 

Assert! handler '= 0 >; ! 

Assert! ( type == prefetch ) I! { type == faultln > > | 

Assert! ' th is- >lock . heldByMe < > ); 

th is- > lock . acquire! ) , 

/* 

* Figure out the range of pages to allocate. 

*/ 

Debug! " 3p ace :: allocate : vT opP age= \ x \ a , vTopPage | 

unsigned int start * this-> vTopP age. j 

void * address = pageToAddr( start j 


Assert! this->isla< address > ) ,• 
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Assert' th is- •> tsl R{ pageToAddr; start -*■ count 


th is- > vTopP age *■ = count. 


* Build the pointer table mappings for the newly allocated pages. 

V 

th is- >bmldMapp ings( start, count, handler i; 
this->lock.release( ) 

/* 

* Do the prefetching of the pages if requested by the caller 

V 

if< type - = prefetch ) { 

Debug( "Space; : allocate, doing prefetch\n" ) ,• 
char • addr = (char * } address 
for ( iat l = 0; i < count,- i + * ) { 

h andler- > f ixFaul t ( this, addr 
addr *= PAGES I ZE ; 


Debug ( " Space :: allocate : returning %x\a" . address 
return; address ) 


* Allocate 'count*' specific pages starting at "base" from the space of type 

* “type", use handler" to manage them. 

V 

vo id * 

Space ; - allocater void • base, unsigned int count. FaultHandler * handler, 
allocat lOnType type ) 

/- 

• Entry debugging and assertions. 

V 

□ ebug( " Space alloc ( basest counted handler \x type:*d ) ■. this=ix\n" 
base, count, handler, type, this ) ; 
if ( count == 0 ) 

returnr O' ) , 

Assert ( handler '*0 ), 

Assert! ( type == prefetch ) || ( type == faultln ) 

Assert ( this- > isln( base ) ) 

Assert ( this->isln< (char *> base +■ ( int ; pageToAddrf count , ) ); 

Assert ( 1 th is- > lock . heldByMe ( ) ) ,- 

this->lock. acquire ! ) : 

/* 

* Figure out the range of pages to allocate. 

V 

Assert ( ( , int)base a Oxlff ) == 0 }.- 

unsigned int start = addrToPaget base ) ,- 
unsigned iat end = start * count - 1; 
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address » pageToAddr( start ); 


Assert ( this->isln( pageToAddr( start ) ) > ; 

Assert ( this->isln( pageToAddr( end ) ) ) 


FIX THIS; it is only • stop-gap solution 


if ( this- > vTopPage <= end ) 

this->vTop?age = end + 1/ 


* Build the pointer table mapping* for the newly allocated pages. 

V 

th is- >bmldMapping*( start, count, handler ) 
th is->lock . release( j / 

/* 

* Do the pref etching of the pages if requested by the caller. 

*/ 

if ( type *3 prefetch ) { 

Debug ( " Space :: allocate -. doing prefetch\n." ) j 
char * addr = (char *} address, 
for ( mt i = 0 i < count,- i++ ) { 

handler- > f ixrault( this, addr 
addr PAGE SI IE ; 


Debugs " Space-. -. allocate; returning *x\n" , address ), 
return( address ) ,- 


* Lookup a fault handler in the per-Space index table. 

* Find an available fault handler index and install the handler 

* if it is not already in the table. 

V 

int 

Space: ; convert FaultHandlerToI ndex( FaultHandler • handler ) 

{ 

Assert( handler '*0 > 

int freeSlot = -l ; 


• Leave slot 0 empty so that pages without a fault handler 

• return the proper value from the handler; ) method. 

V 

for ( int index = 1,- index < MAXHANDLERS.- mdex+ + ) { 

if( f aultHandler [ index J == handler ) 
return) index ) ,- 
if ( ( f aultHandler [ index I ** 

freeSlot = index,- 


9 1 ( freeSlot 


-1 \ ) 


5>t ( freeSlot < MAXKAN D L E RS » ) ,- 

handler ; 
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if' freeSlot. == -l i 
return' 0 < ; 

Assert freeSlot >= 1 , 
f auitHandler ( freeSlot ] = 
return! freeSlot )„■ 


* Build the pointer table mappings for "count' pages starting 
j * managed by 'handler" 


vo id 
Space 

{ 


buildMapp ings ( unsigned iat start, unsigned int count 
F aultKa ndler * handler ) 


* Entry debuggings and issertio ns. 


"start" 


j 


Debug; ' Space ;: buildMapp mgs » start; *x count; *x handler ; ax >\n 
start, count, handler ; .■ 

Assert this->isln; pageToAddr start i ) •; 

Assert this->islaf pageToAddr; start ► count - 1 ) ) > . 
Assert' handler := 0 }, 


* Find an available fault handler index and install the handler 
*/ 

int index = th is->c onver t F aul tH andlerTo I nd*x, handler , ; 

Debug ( " sp ace ; : buildMapp mg s : Using handler index index 


* loop through each page and initialize the pointer table entry for 

* each Allocate new pointer tables as needed. 

*/ 

unsigned int page = start, 

for ( unsigned mt i = 0 .• i < count, ) { 


unsigned int lllndex = ; page > > 7 i s. Cxff; 
unsigned int Hinder = page 4 3x7ff. 


/* 

* Checfc if a pointer table needs to be allocated 

* for this page and get one if it does 
V 

if) this->table[lllndex] . secondLevelPTE = = 0 ) { 

Debug< ' Space ;; buildMapp ings ; allocating a table\n" )• 
Assert) ‘ < this->table [lllndex] . f irstteveiPTE . val id; ) ) ) 


PTE * pt = this-> allocatePointerTablef page > 

) 

Assert ( th is- > t able { 1 1 1 ndex ] . f lr st LevelPTE . val id ( ) ) ,- 

Assert) th is- > t abl s [ 1 1 1 ndex] . secondLevelPTE '-0 ) „• 

Assert) : th is- > table [ lllndex] . secondLevelPTE [ 12 Index] . val id; > ) 


I 


| 
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/* 

* Set the fault handler index in the pointer table entry 

*/ 

this->table [lllndex] . secondLevelPTE [12Index] . handle; index, 3 
j //HOW DO I GET THE RIGHT PROTECTIONS HERE 

pag--+; 

I } 

I } 

/* 

* Allocate a pointer table for a normal space. The table is allocated out 
% of the CPU's heap space 
V 

PTE % 

Space :; allocatePointerTable) int page ) 

Debug) " Space allocatePo interTabl* ( page; Ad )\n" , page ) „• 

Assert; this->isln{ pageToAddr) page ) ) ); 

unsigned int lllndex = {page >> 7) & Oxff, 

void * pPTaddr s ( this- > store >-> allocate ( L > ,• 

Debug; " Space :: alloc at ePo interT able : pPT addr= Ax\ n" pPTaddr ) .• 

Assert; pPTaddr '= 0 }; 

Space * heap = He->heapSpace( ) 

Debug) "Space : : allocatePointerTable : heap = u\n" , heap ) 

Assert; heap ■= 0 i ; 

void % vPTaddr = heap- > alloc ate ( l )• 

DeDug) "Space: ; allocatePointerTable: vPT addr=* x\ n ' vPTaddr ) . 

Assert' vPTaddr '= 0 ); 

heap->map( vPTaddr. pPTaddr, 1 ) 

Assert) * : th is- > table [ lllndex] . first LevelPTE. val id) ) ) 
this- >tabletl Under] . secondLevelPTE = PTE M vPTaddr, 

( this->table[ lllndex] . f irst LevelPTE ). map) addrToPage) pPTaddr ), 3 i ,■ 

//HOW DO WX CHOOSE THE RIGHT PROTECTION LEVEL HERE- 

/* 

*• Initialize the new pointer table. 

V 

ImtSecondLevelPageTablei (PTE *) vPTaddr ) ; 
return; (PTE *) vPTaddr ) ; 

) 

vo id 

Space ;; getPo incerT ablest unsigned int startPage, unsigned int endPage ) 

( 

Debug) " Space :: getPo laterTables ( %x, %x )\n", startPage, endPage ) , 

Assert; NOTSEACHED ); // this routine will die soon. 

} 


( 


! 


I 


I 

1 


! 

j 

I 
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• Map a virtual pa^e to i physical page Crime 
*/ 

vo id 

Space mapr void * page, void • frame ) 

f 

Debugs " Space : : map ( page: >ix frame: *ii ): this = Hx\n " 

page, frame, this 
Assertj this->isln( page ) ) „■ 

Assert.* ’ th is- > lock . heldSyMe ( ) 
th i s- > lock . acquire ( ) 

/• 

* Map the virtual page to the physical page frame 

unsigned int lllndex = ( (YA ) p age ) . f irstLevell ndex< > . 
unsigned int 12Iadex = ; (VA) page ) secondlevelindetf > . 

Debug < "Space- map: llladex^id 12 1 ndex= *d\ n" , Hinder. 12Index 

Assert ,; th is- > t able [ 11 1 ndexj f ir st L e velPTE . v il id < > 

Assert- th is- > table { 1 llndex ]. secondLevelPTE ' - 0 

Asserti 1 th is- > t able [ 1 1 1 ndex ] secondLevelPTE «. 12 1 ndex’. . vai id ■ i. 

this->table[lir ndex] . secondLevelPTE f 12! ndex] . mao t iddrToP lge > page 
//ROW DO I PUT THE PROPER PROTECTIONS RE RE ' 1 ' 

Assert < this- > table [ 1 llndex] . secondLevelPTE { 12 1 ndex] valid! ) ) 

th is->lock . release ( ) ,• 


handler - return the fault handler function (if any) for a given 
i * new virtual address. 

I */ 

! FaultHandler • 

! Space handler ( void • vaddr ) 

Debug ( ” Sp ace :: handler ( addrix }: this=ax\n', vaddr. this > 

Assert ( this-)isln( vaddr ) ) „ 

Assert( 1 this->locfc. heldSyMe^ ) 
this->lock. acquiref ) 

unsigned 11 ix - ( (VA) vaddr ) . f ir st Level I ndex( ) ,• 
unsigned 12 ix = ((VA) vaddr >. secondLevell ndex <) , 

Debug{ "Space : handler : llix=id 12ix=id\n", llix, 12 ix >, 

if< th is- > table (11 ix] . secondLevelPTE -= Q > l 

Debug ( "Space :: handler : invalid Llpte\n" ) 

Assert ( ' this- > table [ 1 1 ix ] . f ir stLevelPTE . val id ( ) ),- 

this->loc)c . release( ) ; 
return! 0 ),• 
f 

PTE • 12pte - s ( th is- > table [11 ixl . secondLevelPTE [ 12 ix] ) ,■ 
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t aul tSandler • theHandler = f aultHandler ( 12pte- >handlerl ndex( ) ],- 

Debugt " Space handler ( %x ) : index: ix, handler: %x\n" . 

vaddr, 12pte->handlerlndex! ) , theHandler 
lock . release ( ) 
returai theHandler ) ,- 

} 

/* 

* Constructor for a kernel (heap) space . It does things differently since 

* there is no heap space for it to get things from (since it is one). 

* Kernel spaces get their state and pointer tables from statically allocated 

* Germ virtual memory. They get physical memory like other spaces. 

V 

(KernelSpace -KernelSpace! store * store, void * base, int length ) : 

( store, base, length ) // arguments to parent constructor 

Debugs "KernelSpace :: KernelSpace! stored! base : ix length : ix )\n", 
store, base, length ); 

Assert! store * * 0 ) ; 


j * Get physical page* for this Kernel Space. 

*/ 

unsigned pages ■ PageCeiling! siteof( KernelSpace j ) >> PAGESHIFT; 

void • pthi* ■ *tor«->allocate( pages 

j Debug! "\tKeraelSpace : : ctor . pthis = ix (id pag»s)\n", pthis. pages ).- 

Assert! pthis ’*0 ); 

I 

i /* 

t * Get virtual pages for this Kernel Space 

: •/ 

KernelSpace * KernelSpaceAllocator ( ) ; 
j KernelSpace • vthis = KernelSpaceAllocator!). 


i ' 

I * Map into Germ. 

! */ 

extern void GermMap! unsigned, unsigned, unsigned ; 

GermMap! (unsigned) vthis >> PAGESHIFT 
(unsigned) pthis >> PAGESHIFT, 
pages ).■ 

/* 

• OK to set this. Side-effect is to call the parent class (Space) 

* constructor . 

V 

this = vthis; 

> 

KernelSpace -. : "KetnelSpacet ) 

( 

extern void Halt< ) ; 

CPUPr int f ( Kernel destructor c*Ll«d'\n." 

Halt! > . 

// GermKernelSpaceDeAllocator ( this ); 
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! 

/* 

* Get pointer tables from pre-ailocated Germ virtual memory net from tne 

* heap. Physical memory is allocated from the Space's state Store ;usc like ' 

* regular spaces. 

*/ i 

PTE * 

KernelSpace: . allocatePomterTable . mt page ) i 

Debug( 1 Kernel So ace alloc at ePo mterT ab le ( page: 'id An" page \ ,- 
Assert( this->isln( page?oAddr( page ) ) ) • 

unsigned int lllndex = (page > > 7 ) s, Oxff.- I 

i 

void * pPTaddr = ( th is-> store ) -> alloc ate! 1 ) ! 

Debug ( ' Xer neiSpace •: getPT : pPTaddr = \x\n' , pPTaddr ) .- 

Assert/ pPTaddr '=0 ). 

extern void * XernelPo mterT ableAiiocate < XernelSpace *, unsigned mt j 

void ’ vPT addr = KerneiPo interT ableAlloc ate : this page j ; 

Debug: SerneiSpace getPT: vPT addr = U' a' vPT addr j 

Assert i vpTaddr '= 3 : .- 

extern void GermMap unsigned, unsigned unsigned 
GermMap. > unsigned) vPTiddr >> PAGESH Z FT 

(unsigned) pPTaddr >> PAGESHIFT 1 , 

Assert! • ( th is- > table [ 1 II ndex ] . f lr st LevelPTE , v a 1 id ( ) ) ) j 

this- > table ( 1 1 Index I . secondLeveiPTE = (PTE *) vPTaddr. 

( th is- > table C 1 llodex ]. f lr stLevelPTE ). map - addr ToP age; pPTaddr , l * . I 

j //HOW DO WE CHOOSE THE RIGHT PROTECTION LEVEL HERE- 


* Initialize the new pointer table. 

V 

I n itSecondLevelPageT able ! PTE *) vpTaddr j .- 
return! (PTE *) vPTaddr ) .- 


vo id 

XernelSpace :■ getPo interT abies( unsigned int startPage, unsigned mt endPioe ) 

Debug ( " Kernel Space ■ : getPo interT ablest *x , U)\a" , 
startPage, endPage ), 

for ■ unsigned int page = startPage,- page < = endPage; oige++ ; ; 

unsigned Hinder = /page >> 7) & Oxff.- 

unsigned 12Index = page i, 0x7f; 


• Check if a pointer table needs to be allocated 

• for this page. 

V 

if( this->table[l Under] . secondLeveiPTE == 0 } [ 
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Debug! " Xer nelSp ace : : getPTs : getting a ?te\Q" ; ,- 

Assert! 1 / this- > table [ 1 II ndex] . t ir stLevelPTE . val id( )) ) ; 


void • pPTaddr = ; th is- > store )-> al 1 oc ate ( 1 ) ,- | 

Debug( "KernelSpace :: getPT : pPTaddr = U\n" , pPTaddr 

Assert! pPTaddr 1 = 0 ) ; 

extern void • Xer nelPo interTableAllocate ( KernelSpace •, unsigned ) ,- 

vo id • vPT addr - Xer nelPo interT ableAlloc ate ( this, page), j 

Debug! " Xer nelSp ace -.; getPT : vpTaddr = \x\n“ . vpTaddr ) , 

Assert! vPTaddr 1 = 0 j ,- 

extern void GermMap( unsigned, unsigned, unsigned ),- 
GermMap! (unsigned) vPTaddr >> PAGESHIFT, 

(unsigned) pPTaddr > > PAGESHIFT , 

1 ) : 


this- > tablet ill ndex] . secondLeveiPTE = (PTE * ) vPTaddr,- 
( th is- > table [ Hinder 1 . f ir stLevelPTE ) .map! 

addrToPagef pPTaddr ) , 3 ) ,- 
//HOW DO WE CHOOSE THE RIGHT PROTECTION LEVEL HERE- 

/* 

* Initialize the new second level page table 

V 

ImtSecondLevelPageTablef (PTE •) vPTaddr ); 

) 

Assert! th is- > table [ 1 1 I ndex ]. f irst LevelPTE . val id ( ) ' ,- 

Assert! this->table[ 1 II ndex] .secondLeveiPTE 1 = 0 ' . 

Assert ( ’ th is- > table [ HI ndex] . secondLeveiPTE (121 ndex] . val id/ ) ) 


vo id 

Space - map/ void * vbase. void * phase, unsigned int count ) 

Assert! ’ th is- > lock . heldByMe > ) ).- 

th is- > lock . acquire! j , 

/* 

* Map the virtual pages to the physical paces. 

«/ 

unsigned vtop = unsigned) vbase * ( count * PAGESIZE ) ,- 
unsigned page = addrToPage( phase ,, 

Debug! ' Space map ( Ax . %x, \d ) : vtop=*x. page=%x\n" , vbase, phase, 
count, vtop, page ) ; 

fot { unsigned vaddr - (unsigned) vbase,- 

vaddr < vtop; vaddr +■= PAGESIZE. page++ ) { 

/* 

* Determine the first and second level page table entries. 

unsigned lllndex = / ■: VA , vaddr ). f ir st Level I ndexi ),- 
unsigned 12 Index = i >' VA ) vaddr ). second! evel I ndex <> .- 

Assert! this->table(lllndex] . f irstLevelPTE . val id( ) 

Assert! th i s- > t abl e [ 1 II ndex ] . seco ndLe velPTE ’ = 0 ) . 

Assert ( 'this- > table [ lllndex] secondLeveiPTE ( 12 I ndex] . valid. 
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* Map the physical page. 

V 

th is- >t ablelllindes} . secondLeveLPTS (12 Index; . map ( page, 3 >; 

//HOW DO I POT THE PROPER PROTECTIONS HERE'' 1 

Assert ( this- > table (il Index] . secondLexrelPTE { 12 I ndex J . val id( ; ) ,- 


this->loc)c - release; }; 


ivoid * 

! Space . alloc ate ( unsigned int count ) 


Assert^ • th is- > lock . heldByKe< > ); 

th is- > lock - acquire ) ; 

Debug( "KernelSpacs allocate* %d ): this=\x\n" count, this ) ,- 


* Figure out the range of pages to allocate. 

Debugf " Space : alloc ate : vTopP ige='^x' n ‘ . vToppage j 
unsigned mt start = th is- > vTopP age .* 
void * address = pageTcAddrf start ) 

Assert this->isln< paaeToAddr* start count - 1 i 
♦/TopPage +■ = count,- 

Debug* " space :: allocate : gett lagPo interTable\n" ) ,- 
this->getPointerTables* start, vTopPage - l ); 

this->lock.release' ) ,- 

Debug' ‘ Space :: allocate *. returning *x\n" . address , 
return, address ); 
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erse.C: Universe :iass lapismentition. Def :r.»a the Spaces accessible 
by a processor at any given time 

SHeader: Universe . c . v 11. 0 37/05/21 15:57:22 russo Exp S 

$Locker: S 


Modification History: 

$Log; Universe. c,v $ 

Revision 11.0 37/05/21 15:57:22 russo 

Console input and private stores. 

Revision 10.11 37/05/06 19:30:43 russo 
added loadContextFor method implementation 

Revision 10.10 37/05/01 10:45:10 russo 
turned off debugging 

Revision. 10.9 37/04/25 13.09 47 russo 

cleaned Log 

Revision 10.5 37/04/22 20:37:50 russo 

added spaceConta in mg method . 

Revision 10.0 37/04/22 07:43:39 russo 

Mew Spaces, Universes and CPU objects work. Finally 1 

Revision 9.1 37/04/11 19:50:07 russo 

initial revision. 


I include 'Assert . h" 

I include ‘Debug. h" 

I include ' md_tune able . h " 
I include 'Store h ' 
itinciude ‘Space h" 

, I include "CPU.h" 

I | include "Universe h" 


original page is 

OF POOR QUALITY 


* Universe constructor. Sets up the mappings discnbed above. 

V 

Universe: :Universe( Universe • where, PTE * pageTable ; 

{ 

/* 

* Entry debugging and assertions. 

*/ 

Debug ( -Universe ;; Universe! *x, 4x }\n", where, pageTable ) 
Assert ( this == 0 ) ; 

Assert( where * = 0 ) ; 

Assert' pageTable * = 0 ) ,- * t 

this = where; 

this->f irstLevelPageTable = pageTable. 
th is- >ker nelSpacs = 0.- 
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th is- >userSpace = 0 ; 


Universe:: Universal ) 


CPUPrintfc " Universe :: "Universe : this = fcx\n‘‘ , this j 
Halt ' ) ; 


* Add a Space to the Universe of addressable spaces on this CPU. 

* This overlays any spaces that are already mapped into the same range 

* of addresses that the new space occupys. 

V 

vo id 

Universe: : addSpacei Space • aSpace ) 

( 

Debug< " Un iver se : : addSpsce ( \x ): this - 4x\n" , aSpace, this ) ,■ 

Assert( aSpace ’ = 0 ) ; 

Assert ( th is • = 0 ) 

/* 

* Hack to figure out which space is being added. This will go away 

* once the Universe REALLY keeps a list of the mapped m spaces. 

V 

if ( aSpace- > startAddress( ) < (void *) TASKLQWADDR ) 
this- > IcernelSpaee = aSpace, 

else 

this->userSpsce = aSpace,- 

/* 

* Copy the first level page table entries in the Space object into 

* the CPUs *real* first level page table. This should be cleaned 

* up to only copy and flush things different than what is already 

* there. 

V 

unsigned iat lcwFrame * addrToFrame( aSpace- > startAddress ( ) ); 

unsigned int highframe * addrToFrame( aSp ace- > endAddr es s ( ) } ; 

Oebugt "Universe :: addSpace : low/high frame: *d/%d\n", 
lowFrame. highFrame ) ,• 

for( iat i = lowframe; i <= high Frame.- i++ ) 

this-> f irstLevelPageT ablet il = aSpace- >table{ i ] f ir stLevelPTE 

/• 

* VERY inefficient way to flush MKU cache, but for now it works fine. 

V 

Wr itePTBO ( ReadPTB0() 

Wr ltePTBl ( ReadPTBli ) ) ; 

Debug ( "Universe :: addSpace : flushed MMU\n" ) ,- 


* Return the Space that an address falls in or 0 if the address 
v space currently in the universe. 

V 

Space • 


m no 
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spaceCont a . r. ir.j , void * address 

' address * — th is - > jcemelspace 
address <= this->icernel3pace 
return! th is- .» kernel Space 


' start Address 
>endAddress • > 


else 


if( address >- this-vuserSpace-JstartAldrass ; 
( address < = th is- > user Sp aca- > » ndAddr • ss < \ \ 

retura{ this- >userSr»ace ) ; 


return! 0 ) 


Universe . : loadCoatextfor ( Thread * newThread ) 
Debug! " Universe :: ioadCor.text For ; Ax i\ 

Assert! this ' = 3 } .• 

Assert' newThread ‘ - ■' , 


' M *P i:a the threads VM context. This involves addins any spaces 

* the thread requires to the Universe. 

* Thl5 should be fixed to only add vhats not already there, or 

* the Universe should be made smart enought to do this 

* Also we need to fix this to remove what this thread do-snt hav- 

* access to. 

* Currently this whole mess is hacked for i single thread. 

Space * space = r.evThrsid- > spaces ' 

Debug ( “ Un l verse : : l o adCoatext For . space - ‘xn.V, space 

i f { space ' s 0 

th is- > addSoace ' space , , 


ORIGINAL PAGE IS 
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* Thread, r: Imp! enter-s a generic thread of execution to build 

* higher level terael processes on top of. 

* Some fields -type, priority) are opaque and ]ust set and read by the 

* icerael . The Space is used to allocate and free memory for the 

* thread. 

* $Header: Thread. c.v 11.3 37/05/24 05:50:18 russo Exp $ 

* JLocker; $ 

V 

/• 

* Revision History: 

* SLog: Thread. c.v S 

* Revision 11.3 37/05/24 05:50:18 russo 

* all but important debugging off * 

* Revision 11.1 97/05/21 23:34:25 russo 

* working on destructor joins 

* Revision 11 0 37/05/21 15:43.34 russo 

* Console input and private stores. 

* Revision 10.22 87/05/12 10:00:18 russo 

* added CermThread class constructor and destructors 

* Revision 10.15 87/05/10 21 09:32 russo 

* Altered to accomidate each thread keeping its own interrupt stack. 

« this makes context switching much easier and much more efficient. 

* Revision 10.12 87/05/01 15:47:43 russo 

j * Revision 10.0 87/04/22 07:24:53 russo 

* New Spaces.. Universes and CPU objects work. Finally' 

* Revision 3.0 37/04/04 14:55:00 russo 

* Multiple threads and timer interrupts. 

* Revision 3.0 87/03/29 15 22:34 russo 

* _nev and ^delete added for memory management. Also, class interrupts work. 

* Revision 1.1 87/02/23 18:11:18 russo 

* Initial revision 


I include "Assert. h" 

I include "Debug. h“ 
♦include " md_tune able . h" 

♦ include " md^const ant s . h " 
I include "Thread. h" 

I include " Frame. h" 
jlinclude "Space. h" 
j I include " Universe . h" 
j I include "CPU h ' 
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* Which h file should these go in? 

V 

♦define C?UPSK_I 0x0800 
♦define CPUPSR~P 0x0400 
♦define CPUPSR~S 0x0200 
♦define CPUPSK~U 0x0100 

♦define CP<JPSR_N 0x0080 
♦define CPUPS8_Z 0x0040 
♦define CPUPSR_F 0x0020 
♦define CPUPSR - L 0 x 0004 
♦define CP'JPSR~T 0x0002 
♦define CPUPSRJT 0x0001 

typedef void (« APrV)(); 

/* 

* Create a new thread, initialize all the internal fields, and pre-push 

* its initial context onto its interrupt stack. 

V 

Thread: : Thread( APFV startAddress , iat • in it ialSt ackPO inter , int argument, 
vnt priority, void • kernellnfo ) 

{ 

/* 

* Enrty Assertions and Debugging. 

•/ 

Debug< "Thread: : Thread( %x , %r) this = %x\n" , startAddress 

imtialStackPointer , this ), 

Assert{ this *- 0 )/ 

Assert( {unsigned) startAddress < = LASTADDRESSABLELOCATION ); 

Assert ( •• unsigned) imtialStackPointer < - LAS TAD ORES SABLE LOCATION ) ; 

/• 

* Allocate an initial frame on the threads interrupt stack. 

V 

char • isp = 6this-> intarruptStack [ stacfcSize] - sizeof( struct Frame ) ; 
this->setIaterrupcStackPomter ( isp ), 

Debugs "Thread :: Thread: interruptStackPointer- *x\a" , 

this-> mterruptStackPointer < ) j 

struct Frame • context = (struct Frame *) this-> interruptStackPointer( ) , 
Debug( "Thr ead :: Thread : imtialContext at %x\n". context ) ; 

/• 

* Load the PC, PSR, MOD, SB, SP and FP register copies with their 

* in it ial contents . 


context- > vectorNuaber =0; // yuck! 

context->pc = (unsigned int) startAddress,- 

context- > psr = (unsigned short) { CPUPSR_I | CP0PSR_S } CPUPSR_U ), 
context- >mod =0; // fixing this could solve the lowmem problem 

// also, it should be loaded with a value 
// that points to something sensible 
* ( uns igned int) mitialStackPo inter; 

= funs igned int) imtialStackPointer,- 


context-) sp 
context- > fp 



the other 


* Pus in argument to the new 

* general purpose registers. 


V 

context- > eO 
context- > r 1 
context- > r 2 
context- > r l 
context- > r4 
context- > r 5 
cont ext- > r 6 
context- > r 7 


argument; 

0 ; 

0 ; 

0 ; 

0 ; 

0 


thread 


rO Zero out ail 


• Fill in the tore ad object fields that the germ -ceecs for 

* the kernels use. 


this->orior ity = priority, 
th i s- > fc er nel I nf o = kernellafo. 
th is - > next = 0 . 
th l s- > 1 a st = 3 , 
th i s - > 1 spas e s - - ; 


* Free up ail the resources owned by a thread and 


Thread: 'Thread ; 




* Should deallocate stacks and other resources here 

* and what if its on a queue somewhere** 

V 

CPOPr int f ( "Thread :: "Thread : this = *x\n', this ). 

Assert i this 1 * 0 ) 

Assert; this ' = Me- > currentThread ( * ); 

char * stackPage = (char *} P age Floor ( th is- > mit lal'JSP , 

CP UP r int f ( "Thread: : "Thread ; stackPage = U\n", stackPage ), 

Space * stickSpase = Me- >un iver se< ) - > spaceCo nta in mg stackPage ); 
CPUPrmtf( " Thread :: "Thread : stackSpace - u\n' , stackSpace ) 

Assert i stackSpace ' = 0 ), 

// stack3pace->dealiocate( stackPage ; . 

Assert! NOTHEACHED >; 


/• 

* Ounp the internal contents of a thread 

* / 

to id 

Thread : : dump' ) 
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struct Frame * £ = (struct Frame *) th is- > mterruptSt ackPo inter (); 


CPUPriat 
CPUPr mt 


f( "dump; (this=%x) frame=ax PC=>sx PSR=^r MOD~%x SP = %x FP-*x\i 
this, f, f — > pc , f->psr, f- > mod , £->sp, f->fp ) ,- 

f( "dump: r0 = *x rl=ix r2 = %x rJ=*x r4 = *x r5 = %x r6 = *x r'=*x\n", 
f- > rO , f->rl , f- > r2 , f->r3, f->r4. £->r5, f->r6, f->r7 ) 


mt 

Thread: ; i sPreemptable ( ) 

{ 

return( 1 ) ; 

} 

/* 

* Threads which run with kernel privledges. 

V 

KerneiThread : KernelThread( APFV st artAddress , int * init lalStackPo inter , 

mt argument, int priority, to id * kernellafo ) 

( st artAddress, init lalStackPomter . argument, 
priority, kernellafo ) 

Debug/ ’’ KerneiThread : : Ker nelThread( ) this = *x\n" this); 

Assert( this '= 3 ) ,• 

struct Frame » f * (struct Frame • ) th is-> interrupts tackPo inter {) ,- 

Debug! ' KerneiThread : : KerneiThread : isp = *x frame = u\a" , 

this-> mterruptSt acfcPointer ( ) , f » 
f- >psr fc= " ( CP r JPSH__u ) ; 

} 

KernelThread : "KernelThread! ) 

CPUP r int f ( " KernelThread :: "KernelThreadf : this = *x\a". this ); 

Assert ( NOT2EACHID ), 

} 

/* 

* THE germ threads constructor, later, try to make sure this is only called 

* once or all hell might break loose. 

*/ 

GermThread : : GermThread( GermThread • where Apr/ st artAddress , 

int * mit xalStackPo inter , int argument, mt priority, 

Toid * kernellafo ) 

( st artAddress, init lalStackPomter . argument, 
priority, kernellnfo ) 

{ 

Assert( where ’=0 ) ,- 
this = where. 

Debug! "GermThread; : GermThreadi ) this = sx\n" , this i; 
struct Frame * f = (struct Frame * ) th is- > mterrup tStackPo inter < ) : 

Debug! ' GermThread GeroThre ad : isp = vx frame = ^x\n", 

th is- > mterruptSt ackPo inter () , f ),- 
f->psr *,= "(CPUPS8_(J 1 CPUPSR_i ) ,■ // system mode, interrupts ol 
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1 / * 

* T ineSl iseTiiraad . i ■ 

* Set up tae timer, wait for it to tick, then set it up agaia. , . forever 

* 

* SHeader: T imaSl iceThread . c , v 11.0 37/0 5/21 15 r 5 5 : C 2 russo Exp $ 

* $ Locker : 5 


i V 

/* 

* Revision History: 

* $ Log ; T imeSL iceThread . c , v $ 

e Revision 11.0 37/05/21 15:55:02 russo 

* Console input and private stores. 

* 

t Revision 10.1 37/05/13 20:29:07 russo 

* initial revision. Split off from K-rnelMain', 

V 


| | include 
j I include 
; 1 include 
j i include 
; I include 
j I include 


"Debug . h ' 
"Assert . h " 
"Except ion . a ’ 

-cpy.h" 

Vectors . h " 

' T imer . a" 


* The time slice interrupt Thread code. 

V 

vo id 

T icker ( int arg } 

{ 

Debug( "Ticker( kx )\n", arg ); 

I nterruptExcept ion * clockTick = new I uter ruptExcept ion ; j ,• 

He- ^ setExcept ion( TIMESLICE_Vector , clockTick ) 

Timer * timer =* new Timer; ) 

Debug ( "T icker : timer = *x\n" , t imer ) .* 

Assert; t imer 1 = 0 ) 

/ * 

* Get our timer initialized and start it running 
*/ 

while; 1 ) { 

timer- > start* 10000 16 ) ,- // Interrupt 16 m 10 seconds. 

// Aren't these nice constants that 
// will come back to haunt us some day. 

clockTick->await( ) 

/• 

* Restart the timer and acknowledge the interrupt. 

•/ 

extern void InterruptAcknovledge; ) .♦ 

t imer- > stop < ) 

Debug; Ticker: Acking interrupt\n" ),- 
I nterruptAcknowledge ( ) 


ORIGINAL' PAGE 15 
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Debug ( Ticker; timer restarted and interr.pt acknowledged\n" 
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Tist .c Page 1 


* Task.c, task class imp lament itiaa , ; 

i 

i 

ORIGINAL PAGE IS i 

OF POOR QUALITY 


• And DONT have the constructor add the initial thread to the scheduler. 

* 

• Revision 10.2 37/04/22 15:41:21 russo 

• hack to add tasks space to list of spaces neccessary far a new thead to run. 

• Revision 10.0 37/04/22 0 7 3 3: 29 russo 

• New Spaces, Universes and CPU objects work, Finally’ 

• Revision 9.1 37/04/05 17.20.49 russo 

• fixed so only one fault handler is allocated for ail the threads stacks 

• rather than one per stack. 

• Revision 9.0 87/04/C4 15:12:41 russo 

• Multiple threads and timer interrupts. 

• Revision 9.0 97/03/29 15:33 54 russo 

• _new and ^delete added for memory management. Also, class interrupts work. 

• Revision 7.0 87/03/25 12:49:41 russo 

• Fault handler hierarchy works, so does interpr oces sor vectored interrupts. 

• 

• Revision 4.1 37/03/08 16:44.24 russo 

* Initial Revision 

*/ 

I include " Debug. h" 

I include "md_tuneable . h" 

I include "Assert, h" 

I include • Space. h" 

•I include " F aul tH andler h " 

I | include "Thread. h" 

It include "Taskh" 

It include " Scheduler . h'* 

[const int ThreadStackSize = PAGESIZE; // how should this REALLY be decided, 
typedef void ( * APFV) ( ) ; 

Task: : Task( Space • space, APfV mit ialEntryPo int ) 


i * SSeader Task.c.v 11.1 37/05/24 17:01:55 russo R X p 5 

! * $ Locker 5 

*/ 

/* 

• Modification History: 

• SLog : Task . c , v S 

• Revision 11.1 87/05/24 17:01:55 russo 

• fixed to use new Space allocate methods. 

• Revision 11.0 87/05/21 15:54:58 russo 

• Console input and private stores 

• Revision 10.8 37/05/12 17:27:12 russo 

• added int iulThread method. 
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{ ! 

/* 

• Initial Assertions and Debugging j 

•/ 

Debug ( "Task: :Task( %x, : thisv%x\n", space, initialEntryPoint , this )„• 

Assert ( this '= 0 )/ 

Asserti space ' = 0 )/ 

Assert; (unsigned int' initialEntryPoint >= TASKLOWADD* ) ; 

Assert; (unsigned int) initialEntryPoint < TASKHIGHADDR ) ,■ ! 

/* 

• Set the space member variables. 

V 

this->space = space,- 
/* 

• Build the fault handler the threads will initially use to fault 

• their stacks in with. (Make sure this is done before any 

• threads are created in this task. ) 

V 

this-) stackf aul t Handler = new DemandZeroFaultHandlar ( ) ,• 

Debug ( "Task: :Task: th is- > st ackr aultHandl er = %x\n" , 

th is- > stackf aul tS andler ); j 

Assert ( this- > stackf aul tHandler ’ = 0 ) 

1 /* 

• Start the initial thread at the initial entry point. 

V 

Debug( "Task: -.Task; calling stertThEead\n" ) ; 
this->threads * 0; 

(void) Task : ; startThread( initialEntryPoint, 3 ) ; 

Debug( "Task: vTasfc: this->thre*ds - *si\a" , this- >threads ) 

Assert; this->threads 0 

Assert ( th is- >threads- > next == 0 ) ; } 

) | 

/• 

• Task destructor. 

• Delete all the threads m the task then the task space. 

V 

Task: : "Ta sk ( ) 

{ 

Pr mtf ( "Task: : "Task: this * %x\n" , this ), 
while ( th is- >threads 0 ) { 

Thread • t = this- > threads,* 
this- >threads - t->next,- 
delete t; 

} 

delete this->space ( - 

} 

/* 

• Return a pointer to the Tasks initial thread. 

V 

Thread • 

Task: : in it i alThr e ad ( ) ' j 
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head~> next = t.- 


thxs->locfc. ielsiie( ) ; 


Debug* “Task : : startThread: returning \x\n“ , 

return* t ) ; 
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rfitsa. 2 : context switching routines 

SHeader: Switch c.v 11.3 37/Q5/2 7 06:52:33 russo Exp S 

$ Locker S 


Modification History: 

SLog ; Switch . c , v $ 

Revision 11.3 37/05/27 06:52:38 russo 

dont turn on debugging unless you mean it 

Revision 11.0 37/05/21 15:43:00 russo 

Console input and private stores 

Revision 10.30 37/05/10 21.09:27 russo 

altered to accomidate each thread keeping its own interrupt stack, 
this makes context switching much easier and much more efficient. 

Revision 10.2 3 9 7/ 0 5/ 36 13 43: 09 russo 

use new On lvar se- > 1 oadC ontext For method. I'm thinking Of including all of 
this as methods af the CPO object. 

Revision 10.20 37/05/04 12:44:28 russo 

fixed problem with disabling interrupts during a switch. 

Revision 10.3 37/04/22 16:00:48 russo 

added from list of thread spaces. This removed all knowledge of Tasks 
from the context switching. 

Revision 10.0 87/04/22 07:24:49 russo 

New Spaces. Universes and CPO objects work, Finally' 

Revision 9.0 97/04/04 14:54:55 russo 

Multiple threads and timer interrupts. 

Revision 3.1 8 7/ C 4/0 4 0 5:10:39 russo 

initial revision 
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I include 'Assert. h" 

I include 'Debug. h" 
t include "CPU . h" 

I include "Thread. h" 

I include "Space. h" 

I include "Universe. h" 


Where, when, and if to disable interrupts here is really leaving me with 
a sick feeling. — Vince 


vo id 

SvitchTo< Thread * aewTbread | 


Debug( "SwitchTo( nevThread %x )\n", newThread | ,- 
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if{ newThread == 0 ) nevThread = Me- > idleThre ad( ) ,- 
Assert( newThread 0 ) ; 

/* 

* Get the current Thread. 

Thread • curreatThread = Ms->currentThread( ) ,■ 

» ifdef DEBUG 

lf( newThread == Me- > idleThread( ) ) 

CPUPrintf; "SvitchTo: from: lx to : IDLE\n" , currentThr ead ) 

else if( curreatThread == Me-> idleThread( ) ) 

CPUPrintf! "SvitchTo: from: IDLE to:%x\n", newThread ) 

el se 

CPUPrintf! "SvitchTo; from: lx to:%x\n", currentThread, newThread ) ,- 

I end l f DEBUG 

Assert! currentThread != 0 ); 

Assert! currentThread nevThread >; 

/* 

* _saveContext( ) saves the current context so that when the thread is 

* restarted it will appear as if _sa veContext ( ) returned 0. 

* The first time it is called it returns the new mterruptStackPo inter 

* to dispatch the thread with. Another side effect is that when it 

* first returns, interrupts are disabled 

V 

extern char • _saveContext ( ) ; 
char • isp,- 

Debug( "Calling _H7eContext( )\n" ); 

if( ( isp = _saveContext ( ) ) =* 0 ) { 

Debug! "SvitchTo: thread *x, rest arted\n" , curreatThread > „• 

Assert! currentThread == Me- > cur rentThr ead( ) j 
return.- 

} 

Debug! "_saveContext returaed\n" ); 

/• 

* Stuff pushed on the stack from here on will be lost upon restart 
*/ 

extern void Dispatch! Thread * ) , 

currentThread-> setlnterruptStackPointer ( isp ) ,- 
Dispatch! newThread ) ,- 
Assert! NOTREACHED ); 

} 

/* ARE INTERRUPTS OK o , */ 


* Dispatch i new thread never to return. The context of the thread to dispatch 

• is assumed to be on its interrupt stack. 

V 

vo id 

Dispatch! Thread * newThread i 
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* Entry Assert ior.s and Debugging. 

V 

Assert ( nevT bread '= 0 

Debug ( "Dispatch! *x ): interrupt staclt will be = Vx\n" 

newThread, newThre ad- > i nter rup tS t ac <P o inter { ; 

/* 

* aemember who were dispatching, load its VM context, and disoatch it. 

V 

extern void _dispatch( char • ). 

Iifdef DEBUG 

newThread- >dump ( | ; 

I end 1 1 

Me- > setCurrentThrsad < newThread 
Me->universe( , - > ioadCo ntext Eor ( newThread ). 

Debug' "Dispatch- _dispatch' *x ,\a" , newThread- > interrupts t acfcPo l nter ' } ) 

_dispatch, newThre ad- > interrupts tackPo inter 
Assert! NOT BEACH2D 
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2 switch . 3 . assembly language context switching routines. 

itch. s , v ll.O 87/05/21 15:43 13 russo Exp $ 


5 Header 
S locker 


Modification History. 

$Log: cswitsh.s,v $ 

Revision 11.0 37/05/21 15:43:13 russo 

Console input and private stores 
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Revision 10.13 87/OS/ll 08:26:38 russo 

was saving the wrong stack pointer for restarted threads in _saveContext . 
I forgot to increment it so as to 'pop-' off the return address 


Revision 10.11 37/05/10 21:09:36 russo 

altered to accomidate each thread keeping its own interrupt stack, 
this makes context switching much easier and mush more efficient. 


Revision 10.0 37/04/22 07.24:39 russo 

Mew Spaces, Universes and CPU objects work, Emaily* 


Revision 9.0 37/04/04 14:55:06 russo 

Multiple threads and timer interrupts. 


Revision 8.0 37/03/29 15:13.07 russo 

_nev and _deiete added for memory management. Also, class interrupts work. 


Revision 7.0 87/03/25 12:43:06 russo 

fault handlers hierarchy works, so does the inter processor s vectored 
interrupt stuff. 


Revision 1.1 37/02/23 17:56:58 russo 

Initial revision 


int _saveContext { ) ; 


Save the context of the current thread and arrange for it to be restarted. 
It is saved by pushing the current context onto its interrupt stack. 

It is saved in such a way that when another Thread switches back to it, 
it will appear as is the call to this procedure simply returned " 0 " . 

Vhen it is really "called" it returns the int errupt St ackPo inter for the 
thread to be dispatched with. Another side effect is that it returns with 
interrupts disabled. 


We assume nothing but the return address is pushed by a C procedure 
call since our compiler passes the first two args in rO and rl 
'Me also assume that rO and rl are volitile registers across 
procedure calls. 


/ 


. globl 
savecontext : 


saveContext 


10:01 198 

7 cswitch.s 

Page ; 

/* 

* Switch stacks to 

this T) 

V 

sprd 

sp rO 

/* 

sprw 

psr , rl 

/* 

b icpsrw 

$(0x200+0x800) /* 

now 

rl.tos ,» 

/* 

sprw 

mod , cos 


movd 

0 ( rO ) , tos 

/• 

movqd 

$0 , tos 

/* 

movqd 

$0 , tos 

/* 

movqd 

$0 , tos 

/* 

movd 

r2 tos 


movd 

r 3 tos 


movd 

r4 , tos 


movd 

r5 tos 


movd 

r6 . tos 


movd 

r 7 , tos 


sprd 

fp, tos 


added 

$0x4 ,r0 

/* 

movd 

rO , tos 

/* 

sprd 

sp , rO 

/* 

b ispsrw 

$(0x200 ) 

/* 

ret 

$0 

/• 


push PSR */ 

PC to restart at ( on top of user stack 


) V 


Rl when restarted */ 


user stack pointer */ 

/* return value * system stack pointer 


* the "real" return */ 


/* 

• void _dispatch( char • mterruptStackPointer ; : 

• Transfer control to another Thread by loading the machine registers 

• from its saved values on its interrupt stack and then "returning" to it . 

• The interrupt stack pointer is activated to do all the work. 

• This can only be called by a thread m kernel mode already. 

i * 

• Register Usage: 

• rO : pointer to the top of the interrupt stack of the Thread to switch to 

I V 

. text 

•globl dispatch 

dispatch: 

bicpsrw $ ( 0x200+0x800 ) / * switch to system stack and make sure •/ 

/ * interrupts are disabled (because */ 

/ * I'm parano id) */ 

lprd sp,r0 

/* 

• this could be a little cleaner if its too slow. The big problem 

• is that the FIRST time this is called, the interrupt stack is 

• already active. 
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ptionc: 5x:?pti5n class implement at iosi . 

$He ider : Exception c.v 11. 0 37/ 05/21 15-42 45 russj Exp S 
3 Locker: $ 


* Modification History: 

* SLog- Exception. c , v % 

* Revision 11. 0 87/05/21 15:42:45 russo 

* Console input and private stores. 

* Revision 10.31 37/05/12 09:31:41 russo 

* use Me - > schdul er () rattier than runQ 

* Revision 10 27 37/05/10 21:03 49 russo 

* altered to accomidate each thread Keeping its own interrupt stack . 

* this aakas context switching much easier and much more efficient. 

* Rev is ion 10.22 37/05/01 11:00:13 russo 

* renamed from Event . c 


ORIGINAL PAGE IS 
.OF POOR QUALITY 


, I include 'Assert. h" 

! include " Debug. h" 

• » include Thread. h” 
j I include "CPU.h" 

: I include "Exception. h" 
1 I include "Frame.h" 

J t include "Scheduler . ii" 


CPUPrintf, " Except ion .: post , %x )\n", frame 
Assert t MOTREACHED ) ; 


SystemExcept ion :: SystemExcept ion< H andler Func t ion theHandler ) 

{ 

Assert; theHandler •= 0 ),- 

th is- > handler = theHandler,- 


SystemExcept ion : : SystemErcept ion t ) 

( 

Debug( "'SystemException: - SystemExcept ion( )\n'" ) 

Assert HOT REACHED ) .* 


vo id 

| SystemErcept ion :: post ( struct Frame • frame ) 

: 1 

i Debug! " SystemErcept ion :: po st %x )\n", frame ),- 

Assert! th is- > handler * = 0 f t 
> * th is- > handler ) ( frame ) . 
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Debug! " SystemErcept ton : post : returning\n" ); 


InterruptErceptxon: : InterruptErcept ion( ) 

{ 

th is- > aw* iter * 0 


: I at errupt Except ion: : 'InterruptErcept ioa( ) 


:// Probably should have Locks in a lot of the stuff here 
vo id 

InterruptErcept ion: : post ( struct Frame • frame ) 

( 

Debug! " I nterruptErcept ion : : post ( %r ) this * %r\n'\ frame, this )/ 
Assert! this * = 0 ) ; 

Assert! frame * * 0 ),- 

Asserti Me->currentThxead( > ' = 0 \ 

/* 

* If no-one is awaiting the event, return to who was interrupted. 

| V 

if! th is- > awa iter =* 0 ) { 

I CPUPrintf' "Un-awaited InterruptException\n" 

j return,- 


* Arrange for the current thread to be restarted where it was 

* interrupted . 

*/ 

Me- > currentThread( ) - > setl nterruptStackPo inter < char •) frame ),- 


* We are not real happy with this, probably the idle thread 
*» SHOULD be enqued, but with the lowest possible priority. 

•/ 

iff Me->currentThread( ) ■= Me- > idleThread( ) ) { 

Assert! Me- > scheduler < ) ‘ = 0 ),• 

Me- > scheduler ()-> add( Me- >currentThreadf ) | ; 


* Dispatch the thread awaiting the event. 
•/ 

extern void Dispatch! Thread • ) / 

Thread • t * this- > awa iter . 
this- > awa iter = 0 ,- 
Dispatch! t )i 
Assert! MOTREACHED ) / 
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* FaultHiadlsrs.c 

* SHeader: FaultHandlers c , v 1 1.5 37/05/24 23 13:21 russo Exp $ 

* JLocicer : 5 

I *' 

/* 

» Modification history; 

* $ L og : FauitSandlers . c , v $ 

* Revision 11.5 37/05/24 23:13:21 russo 

* use two argument space;: map 

* Revision 11.4 97/05/24 16 4538 russo 

* cleaning up 

* Revision 11.1 97/05/24 05:08.20 russo 

* in the middle of re-doing fault handlers. 

* Revision 11.0 37/05/21 15:54:39 russo 

* Console input and private stores. 

* Revision 10.2 37/04/22 20:44 12 russo 

* get space containing the faulting address from the Universe. 

* Revision 10.0 37/04/22 07:39:13 russo 

* New spaces, Universes and CPU ob]ects work, Finally" 

* Revision 9.0 37/04/04 15:12:23 russo 

* Multiple threads and timer interrupts. 

* Revision 30 97/03/29 15:33:40 russo 

* _n.ew and _delet* added for memory management. Mao, class interrupts work. 

* Revision 70 37/33/25 12:49:30 russo 

* Fault handler hierarchy works, so does interprocessor vectored interrupts. 

* Revision 1.1 87/03/19 17:36:12 Johnston 

* Initial revision 

*/ 

I include 'Debug h" 

I include 'Assert . h" 

I include "VM.h" 

I include "Store. h" 

I include " Space. h" 

I include "CPU.h ' 

| include " mi_ tune able . h" 

I include " md^tuneable . h" 


• Common code for all subclasses of FaultHandier . 

*/ 

void 

FaultHandler :: fixFaulc/ Space • space, void • address > { 

/* 

* Eventually this should do the equivalent of the UNIX SIGSEGV 


ORIGINAL' PAGE IS 
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* and terminate the Task. 

V 

Pnatf ( " FaultHandier : tixrault) space . *x address: \x) : this=\x\n” , 

space, address, this ); 

Assert ( NOTSEACHED ); 


/* 

• StoreManager class. 

V 

StoreManager : StoreManager) Store • store ) 

Debug) " StoreManager :: StoreManager ( store . \z ): this = %x\n'‘ , 
store this ) ,- 
Assert ( this ' = 0 ) , 

Assert) store *- 0 ) ,■ 

th is- > storeBeingManaged » store,- 


StoreManager: : ’StoreManager ) \ 

{ 

Debug ( ' StoreManager :: "StoreManager () : this = *x\n”, this ) ; 

Assert { NOTSEACHED ),- 


void 

StoreManager : f ixFault) Space • space, void • address ) 

{ 

Debug ( •" StoreManager :: f ixrault) space: %x addresser ): this = *x\n“, 
space, address, this ) ,- 
Assert) space ‘ = 0 ) 

Assert ( space- > isln( address } ) 

Assert) this ' = 0 ) ,• 

Assert) this->storeBeingManag#d '=0 ) ; 

void a frame = this- > storeBeingManaged- > allocate) l ); 
void • page = (void M PageFloor) iddress 

Debug) "StoreManager :: f lxFault : page=*x, frame=%x\n , ‘ , page, frame ) 
space->map) page, frame ) ,- 

} 

/* 

a Common code for all subclasses of DemandFillFaultHandler ( alloc ateAndMap ) . 

a Demand fill fault handlers are those that, upon a fault, allocate a page 

* from the store, map it into the VH Space, and the fill it with something. 

* what they fill it with depends on the particular sub-class of 

* DemandFillFaultHandler being referenced 

V 

vo id 

DemandFillFaultHandler :: allocateAndMap ( Space • space, void • address ) 

{ 

extern Store * MainStore,- 

Debug) “ DemandFill FH :; allocateAndMap ( space;%x address : *x ): \n" 
address, space ) 
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! 

Asj?:t space ’ = 3 


» Determine the virtual page number that faulted. 

*/ 

void * virtualBase = .void *) Pag-Floor' (unsigned) address ).- 


* Allocate a physical page from the store and 
*/ 


Debug ( " DemandFlilF aultHandler : . al 1 oc a t eAndMap 

void * physicalBase = Ma mStore- > alloc ate r, 1 
Assart, phys ic a. 3 a se •= 3 ) .■ 

space->map( virtualBase, phy s ic al3 a se . L ) ,- 
Debugf " DemandFillFaultHandler : ailocatsAndMap 

virtualBase physicalBase ),- 


map it in. 
getting a page\n' ) 


irirt: U phys.Ax\n" , 


original; page is 

OF POOR QUALITY 


* Constructor for the demand filler class fault handler class that uses 

* the filler class to fill tne ?*ges with . Its only :co is to set the filler 

* member equal to the argument. The filler class member function iillPage 

* will be used by this classes f:xFauit( member function to fill the 

* faulting page after it has been allocated and mapped in. 

Demand? lilerCl assFaultHandler : 

Demand? llierCl assFaultHandler < Filler * filler i [ 

* Allocate space for state information. 

* WE SHOULD CHICK FOR DUPLICATES TO SAVE SPACE AND ALSO DO REFERENCE 

* COUNTING SINCE WE REALLY DONT KNOW WHEN TO DELETE ONE 

V 

Assert this ‘ = D >, 

Debug.- " DemandFillerCI assFaultHandler : Demands il ler ‘Si this=\x\s" 

th is , filler > ; 
th is- > f lller - filler 


/* 

* Destructor for the above demand fill fault handler sub-class. 

*/ 

Demand? illerCl assFaultHandler : : "Demar.dF lilerCl assFaultHandler ) { 

Pnatfi ’‘DemandFlilF aultHandler destructor called' * '\a" ) 

delete filler,- 


I 


/* 

* Demand fill (using a Filler) fault handler subclass fixFaultf) routine I 

* Allocate a page from the store, map it in to the Tasks VM , and the ! 

* call the filler members fillPag*!; member funtion to initialize the page. I 

v ! 

vo id 

DemandF lilerCl assFaultHandler : : f ixF ault i Space * space, void * address ) 

: { 

Debug( " DemandFillerClassFH ::£ txFault t space : addresser); this=*x\n", i 

space address, this ); ' 
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alloc at eAndMap< space, address ; 

* Call the fillers fillPage member function to fill in the page. 

V 

Debug( ‘’DemandFillerCl assFaultHandler f ixFauIt : calling filler\n" 

f lller- > f iilPage( address ). 

Debug ( ” Dema ndF illerCl a ssF au„ tH a ndler ; : f lxFauI t : r etur n ing\ n" ), 


/* 

• The constructore for the subclass of demand fill fault handler that fills 
» the faulting page with zeros. 

*/ 

Dema ndZeroF aultHandler: DemandZeroFaultHandler ( ) 

{ 

/ » 

* Allocate space for state information. 

* WE SHOULD CHECK FOR DUPLICATES TO SAVE SPACE AND ALSO DO REFERENCE 

* COUNTING SINCE WE REALLY DONT KNOW WHEN TO DELETE ONE 

V 

1 Assert) this ' = 0 ) ; 

' Debug! "DemandZeroFaultHandler : : DemandZeroFaultHandler : this = *tx\a ' , 


/* 

* The destructor for the above 

V 

DemandZeroFaultHandler: : "DemandZeroFiultHandler ; ) 

{ 

?rmtf( "DemandZeroFaultHandler destructor called' ,, \n" ) : 

} 

/* 

* The fixFault routine for DemandZeroFaultHandler. This calls the parent 

* classes allocateAndMap( ) member function to get a page and make it 

* addressable Then it simply fills the new page with zeros. 

V 

vo id 

DemandZeroF aultH andler :: f irF ault ( Space • space void • address ) 

{ 

Debug( " DemandZeroFH : : f ixFault< space : %x iddresstx j: this=%x\n", 

space, address, this ),- 
alloc at eAndMap! space, address } ,* 

/* 

* Calculate the base address of the page and fill it with zeros 
*/ 

extern void ClearMemory! void • unsigned ); 

void • virtualBase = ! vo id *) PigeFlooc; (unsigned) address ) .- 
Debug! " DemandZeroFaultHandler :: f ixFault : ClearMemory! U, ^x )\n', 
VirtualBase, PAGESIZE ) .- 
ClearMemory! virtualBase PAGESIZE ). 

Debug! " DemandZeroFaultHandler f ixffault returnmg\ n" )/ 


I 


| 


} 
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CDFFSout loes . z ; routines to deal with setting up a space to toe loaded 
from a UNIX COFF Common Object File Format) image. 

SHeader: COFFRout mes c v 11.4 87/05/27 05:34:13 russo Exp $ 

SLocker: $ 


Revision History: 


* Bevision 11.4 37/05/27 05:34:13 

* debug off 

russo 

* Bevision 11.3 37/05/25 05:41:21 

• debugging on 

russo 

• Bevision 11.2 87/05/24 17:04:33 

• spelling error 

russo 

- Bevision 11.1 37/05/24 16:44 34 russo 

* switched to new Space allocate routines. 

• Revision 11.0 37/05/21 15:34:27 

* Console input and private stores 

russo 

• Revision 10.0 37/04/22 07:38:03 

* New Spaces, Universes and CPU ob} 

russo 

ects work. Finally' 

* Revision 9.0 97/04/04 15:12:17 

* Multiple threads and timer inter: 

russo 
upts . 

• Revision 8.0 87/03/29 15 33:35 

• _new and _delete added for memory 

russo 

management. Also, class interrupts work 

* Revision 7.0 37/03/25 12:49:05 

• Fault handler hierarchy works, so 

russo 

does iaterprocessor vectored interrupts 

• Revision 6.1 37/03/22 17:53:46 

* Initial Revision. 

V 

russo 

1 include ‘ Debug. h" 

1 include "Filler. h" 

1 include " F aul tHandler . h " 

1 include “File.h” 

1 include "Space. h” 

1 include "md^tuaeable . h " 

* include "/usr/ include/a . out h" 


void bcopy ( char * , char *, mt j 
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typedef void (* APFV)<); 


Setup a Space to be demand filled from a COFF image in a File. 


May 28 10:02 1987 COFFBoutinee c Page 2 


IAPFV 

jsetupSpaceFromCOFFImage< Space • space. File • file ) 


• THIS IS A LOT or STUFF TO POT ON THE STACK, IS IT TOO MUCH' 5 

V 

union { 

struct filehdr filehdr,- 
struct aouthdr aouthdr,- 

} in- 
struct scnhdr scnhdr,- 
int filePointer = 0,- 

Debug( " setupSpacerromCOFFImage ( , \x)\n" , space, file ), 
Assert( space '=0 ) ,- 
Assert ( file ?= 0 )/ 


• Bead the file header and check if it's ok (magic 


NS32CMAGIC) . 


Debug< M SetupSpacerromCOFFImage : Beading file header\n“ ),- 
f ile->readBecords( filePointer, (char *) iu. filehdr, 
sizeof( struct filehdr } ),- 

filePointer * = sizeof{ struct filehdr ),- 
if( u. filehdr. fjnagic • = NS32GMAGIC ) { 

Printf( '"SetupSpacerromCOFFImage : Bad Tile Magic Number %r\n 
u . filehdr . f_mag ic ),- 
r«tuxa( (APFV) 0 


* Bead the a. out header from the file, then read the text, data, 

* and .bis section headers and remember the useful bits of 

* information in each. 

*/ 

Debug ( •' SetupSpaceFromCOrrimage : Beading a . out header\n" ■ , 

f ile->readRecords( filePointer, (char * ) tu. aouthdr, 

sizeof( struct aouthdr ) ; 

filePointer += sireof( struct aouthdr ) 

Debug ( “ SetupSpaceFr omcOFFImage : Beading text section header\n" ) ,- 

f lie- >readBecords( filePointer. (char *) iscnhdr, 
sizeof( struct scnhdr ) ) ,■ 

filePointer *= sizeof( struct scnhdr ) ,- 
long textScnPtr * scnhdr . s_scnptr 

Debug ( "SetupSpacerromCOFFImage: Beading data section header\n" ) ,- 
f ile->readBecords( filePointer, (char * ) fcscnhdr, 
sizeof{ struct scnhdr ) ) ,■ 

filePointer += sneofi struct scnhdr ) ,- 
long dat aScnPtr = scnhdr . s_scnptr , 

Debug( " SetupSpacerromCOFFImage : Reading bss section header\n" ); 

f ile->readBecords( filePointer, (char *) tscnhdr, 
sneof( struct scnhdr ) ),- 

filePointer +■= sizeof( struct scnhdr ) .- 
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/* 

* Calculate the addresses and sizes of eaca of the sections 

* The lata and bss sections are assumed to be contiguous and are 

* are treated as one section. 

V 

APFV entryPoiat = (APE"/) u . aouthdr . entry . 

void * textstart = ivotd u . aouthdr . text_st art . 

void * dataStart = (void *) u . aouthdr data_st art . 

Debug* " SetupSpaceFromCOFFImage : textSt art = '* x • dat aS t ar t=*x »ncry= U\a" , 

textstart dataStart, entryPoint 1 .■ 


Assert) 
Assert ' 
Assert ( 
Assert ( 
Assert ( 
Assert ( 


i ( uns igned) textstart % PAGE SIZE) = = 
; uns igned ) teitot art >= TA5KL0WADDR 
( unsigned) textstart < TASKHIOHADDR 
i ( uns igned ) dataStart ^ PAGSSIZE) = = 
* uns igned ; dat aStart >= TASKLOWADDR 
(unsigned dataStart < TASKSIGHADDR 


3 •• ; 

0 } ; 


mt textSize = u, aouthdr . tsize,- 
int textPages = nextSize / PAGSSIZE. 
if ( » texts l ze s PAGE SIZE ) ‘ = Q { 

textp ages-M- 

} 

Assert* uns igned) ( textstart + textSize) < TASKHIGHADDR 
Assert* (char • /entryPoint >= textstart > ; 

Assert ( (char *)entryPomt <= (textstart textSize* 


tat data Size = u . aouthdr . ds ize +■ u . aouthdr . b s ize , 
int dataPages = dataSize / PAGESIZE; 
if ( i dataSize PACESIZE ) • = 0 ; { 

dataPages*-*- .- 

} 

Assert; r uns igned )( dat iStart + dataSize) < TASXKI GHADDR ; 
Debug; ‘ SetupSc aceFr omCOFFImage : texts i ze= ** x , textP ages= sd\ a ’ . 

texts ize. textPages 

Debug* '" SetupSpaceFromCOFFImage : dataS ize= ‘■x , dataPages= d\n" 

dataSize, dataPages 


/* 

* Build a fault handler for the .text section. 

V 

□ebug( "SetupSpaceFromCOFFImage: Building .text section filler\n" }; 

CO FF Sect lonF iller • textFiller = new COFFSect lonF iller * file, textstart, 
texts ize, taxtScnPtr ); 

Debug; "textFiller = %x\n" , textFiller ) , 

Assert! textFiller " = 3 ) ,- 

Debug* " SetupSpaceFromCOFFImage ; Building .text fault handler\n" ); 
DemandF illerClassFauItHandler • textFaultHaadler = 

new DemandFiilerCl aasFaultHandler ( textFiller }.- 
Debug* " textFaultHandler = *x\n', textFaultHandler ) 

Assert! text F aulcHandler * = 3 ). 


/* 

* Allocate the pages for the text section and install the 


I 

I 

I 


I 
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* fault handler created above. Fault handlers should probably I 

* be deleted by the Space destructor when it is fully implemented, 1 

* but they also may need to be deleted when they are no longer j 

* needed. For example, if all the data has been faulted in. the 

* fault handler to load it is no longer needed since all of the I 

* data pages will have there fault handlers replaced by some form ( 

* of swapping fiult handler if their memory is reclaimed. Granted, 

* if a clean paje is chosen for replacement, the original fault I 

* handler will at ill be needed to reclaim it when needed. ] 

*/ 

Debug: "* SetupSpaceFromCOFFImage Allocating the text pages\n 1 ) ,■ j 

char • text = ^icaar *) sp ace- > alloc ate ( textstart, textPages, 
textFaultHandler, faultln j; 

Debug! " SetupSpaceFromCOFFImage : text = \x\n“ , text ) 

Assert ( text 1 = 3 ) .- 

/* 

* Build the fault handler for the .data and . bss section. 

*/ 

Debug* ■SetupSpaceFromCOFFImage: Building data and bss filler\n" ),- 

COFFSect lonF iller * dataFiller = new COFF Sect 10 nF il 1 er ( file, dataStart, 

dataSize, dataScnPtr ); j 

Debug: "dataFiller = %x\n" , dataFiller ). 

Assert* dataFiller *= 3 ),- 

Debug* " SetupSpaceFromCOFFImage : Building data .bss fault handler\n’ ); 

DemandFillerClasaFaultHandler • dataFaultHar.dler = 

new DemandFiilerCl a s sFaul tH a ndler * dataFiller ); 

Debug* '"dat aFaultHandler = *x\n'' , dat aFaul tH andler ) . 

Assert* dat aFaultHandler ’=0 ); 

,/* 

* Allocate the pages for the .data and bss sections and install the 

* fault handler created above. 

V 

Debug! "SetupSpaceFromCOFFImage: Allocating the data pagesyn" ) .- 
char • data = 'char •) space-) allocate* dataStart. dataPages, 
dataFaultHandlex faultln i 
Debug ( "SetupSpaceFromCOFFImage. data = Kx\n" , data 
Assert! data ' = 3 ) 


/• 

* Return the entry point address for this Space. This value is 

* passed to the Task constructor when it is called. 

V 

Debug* "SetupSpaceFromCOFFImage: returning *x\n“ , entryPoint ) ,■ 

return* entryPoint ) ,■ 


/* 

* Constructor to fill in the internal field* of a COFF section 


V 

COFFSect ionF iller : : COFFSect lonF ill er 

l 


File * file, void * start 
int size, long location ) 


/* 

* Entry Assertions and Debugging. 


f iller . 


I 

j 

; 

i 

i 
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Debug ( "COFFSect lonFiiier COFFSect lonFiiler ( ix, id , kx > thi 
start, size, location, tins 
Assert.' this ' * 0 ■ ; 

Assert i file 1 » 0 


initialize the internal fields. 


Assertf {. uns igned ) start % PAGE 3 1 2E ) s- 0 | 

this->file = file; 
th is->secz lonStart = start; 
th is- > sect lonLength = size,- 
th is-> t ileLocation = location,- 


ORIGINAL PAGE is 
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Destructor for GOFFSection fillers. DONT delete the file here tl think’ 


Prmtf "COFFSect lonF iller Destructor calied ,, \n' 


* round faulting address down to its page number. 

V 

Debug* "COFFSect lonFiller : : f illPage( %x)\n" , f aul t ingAddress > .• 

void * virtualBase = : void * ) PageFloor' ^unsigned) fault ingAddress ); 

Debug( ' COFFSect lonF iller : ; f lllPage : virtuaiBase = kx\a'% virtualBase 

* copy in the pages data from the COFF section. 

*/ 

int whichOffset = ' mt ) virtualBase - inti sect lonSt art ,- 

long source = f ileLocation * whichOffset, 

• SHOULD FILE LOCK THE HEADS AND WRITES HERE” 

• ALSO, WHILE IN THINKING ABOUT IT. WHO DELETES THE FILE” 

* I THINK CGFFSpace should be a subclass of task space and its destructor 

* should delta the coff file. 

V 

Debug ( "COFFSect ion.F iller f illPage : offset kx in section addr = , kx ) \n" 

whichOffset, virtualBase ),- 

iff source > ( f ileLocation + sec t lonLength ) > { 
extern void ClearMetaor y( void *, int ) ; 

Debug { " COFFSect lonF iller :: f illP age : filling with zeros\n" ) ,- 

ClearMemoryf virtualBase, PAGES I 2E ) ,- 

else { 

Debug ( " COFFSect. lonF iller .: f illP age : copying in page\n’ ),- 

f lie- >readRecords ( source, (char *) virtualBase PAGESUE ) 


May 23 10:02 1587 COFFRout ines . c Page 6 


Debug ( "COFFSect lonFiller -. fillPage: returning\n" 


* Copy 'count 1 bytes from "from" to "to". Replace this with a nicely optimized 

* assembly language version eventually, 

V 

vo id 

bcooy( char * from, char * to, int count ) 

{ 

while( count ' = 0 } ( 

•to++ = *from++; 
count — 
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GoOnixne 3: First, •- language routine nailed ay each procesor after doing 

initialization Spin in a iocp and wait for an inital thread 
to be added to the scheduler Once one is there switch control 
to it and pray we NEVER return. 

ORIGINAL PAGE IS 
OF POOR QUALITY 

* Revision 1L 0 97/05/21 15:42:49 russo 

* Console input and private stores. 

* Revision 13.43 37/05/13 06:35:44 russo 

* took out setEscept ion for jJDN^OLE^Yector 

* Revision 1041 97/05/12 17:37:12 russo 

* set up to be called by the germs initial threid. 

* Revision 10.35 37/05/02 15:4254 russo 

* split into a germ half (this) and a kernel half KernelEntry i 

* Revision 10.23 87/04/30 16:09:21 russo 

* build a private I nterruptEicept ion for the clock and and mtial thread to 

* handle the clock interrupt. Dispatch that thread. It will await, the 

* event mechanism should restart the idle thread, when the interrupt 

* occurs the clock thread should run again, reset the clock, and await the 

* interrupt again. 

* Revision 10 15 37/04/28 10:35:50 russo 

* build a default exception handler and install it for all vectored exceptions. 

* Then re- install the proper ones for those we care aoout 

* Revision 10 0 37/04/22 07:24:41 russo 

* New Spaces, Universes and CPU objects work, Finally! 

* Revision 9.0 87/04/04 14:54:48 russo 

* Multiple threads and timer interrupts. 

* Revision 8.0 87/03/29 15:22:17 russo 

* _new and _delete added for memory management. Also, class interrupts work. 

* Revision 1,1 87/02/23 18:11:18 russo 

* Initial revision 

V 

I include ‘Assert fa" 

I include 'Debug . h*' 
t include md_r uneable . h" 

I include "Thread. h M 
I include ' Except ion. h“ 


SHsader: GoOnline.«,v 11.1 37/05/21 16:57:57 russo Exp S 
3 Locker : 5 


Revision History: 

Slog: GoOnline.c,v $ 

Revision 11.1 37/05/21 16:57:57 russo 

CPU member changed names 
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I include "Lock h" 

I include Space . h“ 
lioclude 'Store. h** 

(include "Universe. h" 

| include ‘CPU h" 

| include ‘Vector s . h" 

/* 

* Default exception handler until the kernel installs what it wants. 

• There SHOULD be no exceptions until the kernel is running and decides its 

* ready. 

V 

extern void Uncaught( struct Frame * ) 

SystemExcept ion Def aultExcept ion( Uncaught ) ,■ 

/• 

% A Lock to arbitrate kernel creation, and a pointer to the initial kernel 
% Space. The pointer is set by the processor that creates it and shared by all 
% processors. 

V 

static Lock kernelLock,- 

static lernelSpace * Iait iallernelSpace * 0 
/* 

• The c language entry point for the Germ thread. 

*/ 

vo id 

GoOnline( unsigned int ID ) 

{ 

/* 

* Say hello Grade. 

*/ 

Assert( Me ’= 0 ); 

Pnntf< 'Processor %x is online ID*%x)\n“ , Ke->id(), ID j; 

/• 

* Sanity checks of the boot code. 

*/ 

Assert/ Me->id() == ID ); 

Assert ^ Me- > un iver se ( ) ‘=0 ; 

Assertf Me- > idleThread( ) '= 0 } 

Assert ( Me- > currentThread( ) '=0 ) 

Assert{ Me- > currentThread( ) == Me- > idleThread/ ) ); 

Assert( Me- > scheduler ( ) == 0 ); 

Assert ( Me- >threadToDelete( ) == 0 )i 
Assert ( Me- > pr ivateStore ( } ’= 0 ) ,• 

Assert( Ke->globalStore< ) '= 0 ) ,• 

extern int InterruptsDisibled( ) ,• 

Assert ( Interrupt jDisabledl ) ) ,• 


C 


% Install the initial (default) exception handlers Again, no 
* exceptions should happen until the kernel is executing. 

*/ 

Debug/ "GoOnline: installing default (panic) except ions\n" ) ; 


build it. 
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int i = 1, l < M -.Laa e 1 0 : V e s 1 3 r e d E ? p : : o a a • 

Me- > setExcept ion; i, iDe f auitExcept oon 


* Test to see if the kernel has been built yet and. if not. 

* If it has been ouilt. someone else has been here first. 

* la this case just ski? the call, release the lock- and continue. 

kernel Lock . acquire' ) ; 

if( I a it laiKernelSpace == 0 t { 

Debug* "GoOnline building I nit ialKernel3pace\n" ) ,* 

/* 

* Build the initial Kernel, 
extern char • VirtualPr ivateMemory ,■ 

char * kernel3zart = VirtualPr ivateMemory * 0x10000 
lait iiIKerneiSpace = new Kernel Sp ace * Me- >gl obaiSt ore ) , 
kernel St art /* MA>IKi RMSLMEM */ 
cnar * TASKLGWADDR - kernel S t art } .• 

Debug. "GoOnlme: Initial Kerr, el - tsvn' InitialKernelSpace 

Assert* InitialKernelSpace '=0 \ ; 

} 

kernel Lock . release i ■ 
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* Add the kernel space InitialKernelSpace. to this 

* processors Universe. Also set the heapspace instance 

* variable in the CPU object. 

V 

Debug* "GoOnline- Adding initial kernel lx to universe\n‘, 
InitialKernelSpace 
Assert* InitialKernelSpace '=0 ) .- 
Me- > un iver se adds p ace * I a it laiiCer nelSpace 

Me- > setHeapSpace InitialKernelSpace . 


3y the time everybody gets here there is a kernel (heap) space 
available to allocate things out of and added to the processors 
Universe. All initial set up should also be completed. 

All that* left do is to turn the current Tread over to the kernel 
by calling KernelMa inf , who will install ail the exception 
handlers and do other kernel in 1 1 ll 12 at 10 n things, then start 
dispatching threads. 


extern void KernelMain* unsigned int 

Debug* "GoOnline: Calling KernelMa in\ n" >; 
KernelMa in* ID „■ 

Assert* HOT REACHED 
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/* 

♦ Except ionHa r.dlers . c . Various exception handling routines. 

* 

* SHeader : Except lonHandlers . c , v ll.il 87/05/27 05-05:13 russo Exp $ 

* Jloclter : $ 

V 

/* 

* Modification History: 

* SLog; Except ion.Han.dler s . c . v $ 

* Revision 11.11 37/05/27 05:35:13 russo 

* debug off 

* Revision 11 10 87/05/26 21:35:25 russo 

* switched the switch to if. ..then. . .elseif since the compiler 

* seems to be totally hosed. 

* 

* Revision 11 9 87/05/26 07:09:10 Johnston 

* Debugging on. 

* Revision 113 37/05/26 26:24.52 johnston 

* Made ABTTrap use PaaicPnatf instead of CPUPrmtf . 

! * Revision 11.7 87/03/24 22:32-25 russo 

! • attempt to solve the case statement wierdness. 

! * 

! • Revision 11.4 37/05/24 06:13:23 russo 

! * trying to figure out whats happening with KILLTHREAD 

* 

* Revision 11 2 37/05/24 05:09:52 russo 

* fixed calls to fixFault. 

* 

* Revision 11.1 57/05/21 16:50:45 russo 

* switched ways of deietemg threads 

* Revision 11.0 87/05/21 15:54:34 russo 

* Console input and private stores . 

* 

* Revision 10.40 37/05/17 14:04:43 russo 

• added terminating threads to the per-cpu delete queue. 

* Revision 10.37 87/05/16 12:45:38 russo 

• renamed from boot/Tr apCatchers . c 


•/ 

t include 

"md_tuneabls . h” 

1 include 

"Debug . h" 

1 include 

■•.Assert . h" 

» include 

“VM ■ h" 

1 include 

" FaultHandler h" 

1 include 

" S VCs . h" 

t include 

"Space . h" 

1 include 

•'CPU. a " 

1 include 

"Thread . h" 

1 include 

"Task.h ' 

I include 

‘Frame . h'* 

1 include 

•Scheduler . h" 
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extern void Halt,')/ 
void 

DumpFramec struct Frame • frame ) 

{ 

CPUPrmtf ( "DumpFr ame ( %x) : PC=%x PS8=%x MOD=%x Vector = ad\n" , frame, 
frame->pc, frame->psr, frame- >mod , fr *me->vectorNuaber )/ 
CP0Pnntf( " r(0-7]a 4x : \X : U \X : *X : *X : %X Sp=%X fp=*X\n“ , 

frame->r0, frame->rl, frame->r2, frame- >r3, frame->r4, 
frame->r5, fram#->r6, fraae->r7, frame- >sp, frame- >fp ) ,- 

} 

void 

ASTTrap( struct Frame * frame ) 

{ 

/* 

* Grab the MHU registers ve*ll need. 

•/ 

unsigned msr * SeadMSB( ) ; 
unsigned eia = 8eadEIA( ) , 

//» ifdef DEBUG 

PamcPrintf ( "Abort Trap: eia=tx msr=%x pe = %x psr=%x\n” , 
eia, msr, frame->pc, frame->psr ),- 
//DumpFramef frame ),- 
//{(MSB *) imsr )->printf { ) ,■ 

// fendif 

/* 

* Get the faulting address from the EIA 
*/ 

unsigned address = ((EIA) eia ). address* ) ,- 
/* 

e Get the Space containing the faulting address 

* wierd things will happen if mapping of 'Me‘ and the Universe 

* is not set up yet. We really shouldn't put ABTTrap m the trap 

* table until were ready. 

•/ 

Space * fault iagSpace = 

Me->univer se( ) ~> spaceCoat a ining ( (.void *) address ) ; 

Debug( "ABTTrap: faulting space * %x\n” , fault iagSpace ) .• 

Assert{ fault ingSpace '=0 ),- 

/* 

* Handle the fault, (first, cheek for conditions we don't understand.) 

*/ 

Assert/ ‘((MSB) msr ) . BPTError ( ) ); 

Assert( '((MSB) asr).BPR() )/ 

Assert( '((MSB) msr ) . BPTBeadError ( ) ),- 

Assert ( '((MSB) msr ) . BPTSt atError ( ) ); 

Debug ( "ABTTrap: Mode of fault: *s.\n", 

((EIA) eia). txPTB( ) ’ "user" : "supervisor" )/ 

/* 

* The first thing we do is see if we faulted on an address in a Space 
*» that has grown. If this is what happened, then all we have to do 
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* either add sc^.e tew entry 3 to the first level cage table it the 

* "inverse or invalidate the cached :iM r J descriptor. 

*/ 

1C. f suit mgSpace-> isVal idi (void * ; address , 1 i 

Debugf "ABTTrap: faulted on a valid iddress\a" ). 

Me- >umvers 3 i )->addSpace< f auit inaSp ac e ) 

// add space also has the nice feature of flushing 
// the MMU Cor us and adding any new PTEs. 

Debug. ‘ABTTrap: retry ing\ n\n ' ) 

return, 

J 

/* 


* If it wasn’t already valid, get the fault handler 

* to fix the page. 

FaultHandler * theHandler = f aul t mgSp ac e- > h a ndl er 1 
ift theHandler == 0 j { 


( if there is one 1 
vo id • ; address ) .■ 


P anicPr mt f 
Halt; ; ; 


he future this should just xil 
fault with no handler 1 '. 


the thread t ask"* ) 


* Call the Space fault handler and return. This results in 

* retry ing/rest art mg the instruction which caused the fault. 
♦/ 

Deougf "Calling handler ( *axi ( space: addr e ss u ) \ n " , 
theHandler, f aul t ingSpac e , address >. 
theK andler- > f lxfiult : fault mgSpace, void • ' address t- 

Debugf "ABTTrap: Handler returned - retry ing\ n\ n ‘ >/ 


void Touchi int x ) {J 
typedef void ( * APFV)() 


vo id 

SVCTrapf struct Frame 


frame ) 


* This is all a bit of a hack until the "into the kernel" object calls work. 

V 

{ 

Debugf "SVC Trap\n" ); 
lifdef DEBUG 

DumpFramef frame >; 

I end if 


Debugf "SVC(*d) USP=*x\a" , frame->rO, frame->sp j 
char * ap = (char *) (frame->sp * 4)/ 

mt svc = frame->rO.- 

iff svc == PRINTF_SVC ) { 
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else 

} 

else 
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// should fault if needed (what a k I ~ ‘ I $ hack!} 
Touch ( ** (char **)ap ) ; 

CPUPrintfi • ( char **)ap )/ 

lf( svc == KILLTASK_SVC ) ( 

CPUPr intf ( "KillTask SVC Called\n" , 

Hal t ( ) ; 

iff svc == El LLTHREAD^SVC ) { // terminate your own execution 

CPUPrmtff "KillThread SVC Called\n" 

Assert) Me- > currentThread ( ) *= 0 ); 

Assert! Me- > currentThreadf ) *= Me-> idleThread( ) >; 

Assertf Me- > thr eadToDelete ( ) == 0 ) ,■ 


/• 

* Set the current processors Thread to delete 

* to the current Thread . 

• The idle Thread cleans up and deletes the exiting 

• Thread, so switch to it. 

*/ 

extern void SwitchTof Thread • ).• 


Me- > setThreadToDeletef Me- > cur r entThre ad ( > ) ,■ 

CPUPrintff "KillThread set thr e adToDel et e\ n " > ,• 

SwitchTo; 0 j ,• // relinquish the CPU 

Assert ( MOTREACHED ); 

) 

else iff svc == STARTTHREAD_SVC ) { 

Debug < "S7ARTTHR£AD_SVC at \x\n", • ; int * > ap ; 

Debugf "currentThread * U\a" Me- > cur r entThre ad t ) 
Assertf Me- > currentThreadf ) '= 0 

Task * th is? a sk = 

Task *) . Me- >current?hr eadt >getKer nellnf o ( ) 

1 Debugf "thisTask = U\a" thuTask • , 

Assert: th isT ask " = 0 ); 

Aprv argl = SAPr; *) ap, 
int arg2 = * < mt * ) [ ap + 4 

Debugf "call startThreadf \x , *x>\n*' argl, arg2 ' .■ 

Thread * newThread = th isTask- > st artThread ■ argi.arg2 
Debugf "newThread = *x\n‘" , newThread ) 

Assertf newThread • * 0 >; 

Assertf Me- > scheduler f ) '=0 ,, 

Me- > scheduler f }-> add( newThread , „■ 

} 

else { 

CPUPnntf( "Invalid SVC (%d) Called\n" . frame->rO ) 
Halt ( ) ; 

J 

frame- > pc + + ,■ 

Debugf "SVC: set new pc to %x\n" , frame- >pc ) ,• 


vo id 

Uncaughtf struct Frame * frame } 

t 

CPUPrintff "Uncaught vectored exception ' hxi\n" , fr ame- > vect orNumber ). 


i 

! 


1 

! 

i 


! 


1 


! 
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DumpFrame frame i 
Hale, | 


* Be sure to call X at erruptAcltnowl edge for strays if we ever do anything but 

* Halt ( ) . 

V 


/ * 

* Catch all exceptions and rs-direct them to the proper handlers. 

* This should be an Exception handler in it own right’. 

*/ 

vo id 

Except lonCatcher ( struct Frame * frame ) 

Debug ( " Except lonCatcher ( *x ): vector «\n'' , frame, 

f r ame- > vec tor Humber > • 

Assert, fr ime- > vect orHumber >= C i ; 

A.ssert; frame- > vect or Number < Me- ' numberOfVectored£xceptions( ) . 
Exception * exception = Me- > exc ep t ion ( i r ime- > vectorNumber ). 

Debug( "ExceptionCatcher : exception = U\ti' exception 

i ifdef ASSERT 

if{ exception == 0 ) ( 

DumpFramef frame <; 

Assertf exception *= 0 )/ 

} 

• endi f 

except ion- >post( frame ) ,• 

// post worries about context savemg and restoring if 
// neccessary or just calling a subroutine otherwise 
// It also worries about which stack to use, etc., 
j Debug ( 'ExceptionCatcher: post ceturned\n" 

|! 
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CHAPTER 1. 
INTRODUCTION 


Workstations provide good interactive computing environments that have consistent 
user response times and support many devices suitable for interactive work including bit 
mapped displays, mice, and keyboards. Supercomputers supply large amounts of sequential 
and vector computing power. However, they do not provide a cost effective interactive 
environment. This thesis introduces the Cross-Architecture Procedure Call, a software archi- 
tecture that allows applications to exploit both systems. Cross-Architecture Procedure Calls 
(or CAPCs) combine virtual memory, high speed networking, and compatible data represen- 
tations to accelerate an application’s computations without modifying its code. CAPCs allow 
workstation applications to use, on a demand basis, faster or more expensive processors as 
compute servers so that each of an applications functions can be executed by the most 
appropriate processor. 

Workstations offer a number of advantages to a centralized timesharing system. A net- 
work of workstations is more resistant to complete failure than a single system. (However, 
the larger number of components increases the probability of partial failure.) Failure of a 
single workstation usually does not prevent other workstations from functioning. Experimen- 
tal software, frequent (or unexpected) reboots, and different operating systems are easier to 
manage with a connected network of workstations. The incremental costs to enhance a net- 
work of workstations are small. 

Several costs offset these advantages. A system administrator must deal with many 
workstations instead of a single, central machine. System resources, which used to be 
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centralized and easily managed, are now distributed across many systems. One resource lost 
in the transition from a centralized system to a network of workstations is the sequential pro- 
cessing power of the large timesharing CPU. Many tasks do not require the high CPU 
bandwidth available in the centralized system. However, some tasks do require this 
bandwidth; moving these tasks to workstations causes unacceptable increases in their execu- 
tion time. 

The most common way to reduce or eliminate this increase in execution time is to ship 
the entire application to a supercomputer. This batch-oriented technique does not exploit 
the interactive features of the workstation. 

Another way to decrease execution time is to restructure sequential applications into 
concurrent applications and then run them concurrently on many processors. Processor 
configurations range from tightly coupled systems sharing common memory to loosely cou- 
pled systems that communicate over networks such as Ethernet [64]. The tightly coupled 
systems provide a centralized multiprocessor environment. They do not offer the same set of 
interactive tools available on a workstation. Some systems automatically restructure sequen- 
tial applications to execute concurrently on their many processors [11,42]. Other systems do 
not restructure the application; the user must manually convert sequential applications to 
concurrent applications. 

Loosely-coupled systems can provide large amounts of processing power. At this time, 
however, network communications times are dramatically slower than local memory refer- 
ences. This communications overhead affects the choice of algorithms. Algorithms that gen- 
erate less traffic between systems replace simple and fast algorithms that work well in 
tightly-coupled systems (with low communications costs). These replacement algorithms 


3 


may increase the processing demands of the application while reducing the network traffic. 

Remote Procedure Calls provide a mechanism to execute subroutines on remote 
loosely-coupled processors [16,66]. Applications can be partitioned so that CPU-intensive 
routines execute on the supercomputer and other routines execute on the workstation. RPCs 
have several restrictions that affect how an application is partitioned. RPC client and server 
processes do not share the same address space. Thus, pointer-based structures do not 
transfer well to a RPC environment and routines on different processors can not share the 
same global variables. All communications between routines must be through the argument 
list. RPC systems require stub routines and special compile-time operations to generate 
instructions to transfer control between the client and server systems. These factors affect 
how an application can be partitioned in an RPC environment. 

Programs often spend large fractions of their execution time in small sections of their 
code. This is often paraphrased as the 90-10 rule: programs spend 90% of their time in 10% 
of the code. Sometimes, performance can be improved by selecting more appropriate or 
efficient algorithms. In other cases, the algorithm in use is already optimal. In these cases, 
the only way to make that section of code execute faster is to place it on a faster processor. 
Often, this 10% of the code is contained within several subroutines. Therefore, these subrou- 
tines should be moved to a faster processor. 

This thesis proposes a software architecture for executing programs in an environment 
with workstations and supercomputers. Applications can exploit each processor’s particular 
features. Interactive portions of an application can execute on the workstation. CPU- 
intensive portions of an application can execute on the supercomputer. This architecture pro- 
vides a standard process model — an application existing in a single address space. Our 
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architecture usually does not require any restructuring of applications programs. 

Our new architecture partitions applications between workstations and supercomputers 

f 

while meeting the following criteria. These criteria reflect our goals not to require restructur- 
ing of applications and to exploit the features of both workstations and supercomputers. 


• The user need not restructure or recode his applications. 

• The programmer can specify an application’s partitioning. Changes to this par- 
titioning do not require changes to the application source code. 

• Interactive tasks execute on the workstation. That is, the workstation is not 
used as a simple terminal to submit jobs to the supercomputer. 

• CPU-intensive tasks execute on the supercomputer. 

• Optimization techniques, such as vector operation and parallel operations, 
specific to certain architectures are still useful for code segments executed on 
those architectures. 

• The compilers for each system need not be modified; a modified loader combines 
the output from the respective compilers into an executable file. 

• The operating system resolves issues of control transfer and data transfer 
between systems. 


1.1. CLASP Overview 

This thesis proposes the CLASP software architecture. CLASP, an acronym for Cross- 
architecture Address SPace, implements a new foundation for the traditional process model. 
This new foundation allows heterogeneous CPUs to share the virtual address space of a pro- 
cess. Tasks within the application execute on the CPU most appropriate to their needs — 
interactive response, large amounts of CPU bandwidth, vector processing. CLASP identifies 
a level of homogeneity necessary to implement this sharing. It also mitigates dissimilarities 
between the processor architectures such as register sets and stack frame formats. CLASP 
makes these differences transparent to the programmer. 
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CLASP allows heterogeneous CPUs with different performance characteristics and 
potentially different instruction sets to operate in a single address space. This allows portions 
of an application to be executed by the most appropriate processor. Where other research 
efforts have augmented standard addressing schemes to provide remote addresses, CLASP 
makes a single address space accessible to multiple heterogeneous CPUs [78]. A novel aspect 
of the CLASP architecture is the inclusion of instructions for different processor architectures 
within the same address space. These instructions are placed in different regions of the 
address space. 

Our architecture introduces a new control transfer mechanism, the Cross Architecture 
Procedure Call (or CAPC). Like the Remote Procedure Call (or RPC), the Cross Architec- 
ture Procedure Call transfers control between processors. RPCs introduce new calling 
sequences into the application code to transfer control between processors. CAPCs do not 
modify the subroutine calling sequence in the application code. In CAPCs, both local and 
remote subroutine calls use the standard subroutine call and return instructions. The 
CLASP kernel detects calls that refer to remote subroutines, packages arguments, and 
transfers the control thread to the remote processor. When a subroutine is moved from the 
local processor to the remote processor, the CAPC system does not require any changes to 
the source or compiled instances of procedures that invoke the migrated subroutine. 

Applications are prepared for this architecture by the new CLASP loader, which links 
separately compiled routines into a single executable image. This loader recognizes the 
different object formats for various processor architectures and resolves the cross- 
architecture references. It provides the operating system kernel with the information neces- 
sary to detect control transfers (e.g., procedure calls and returns) that cross architecture 
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boundaries. Routines that execute on specific architectures are compiled for those architec- 
tures. Some frequently called routines (e.g., sqrt()) can be replicated. Duplicate copies of 
these routines, each compiled for a different architecture, are loaded into the executable file. 
Calls to any of these routines can be directed to the local instance of that routine, saving the 
network overhead of a remote call. The loader chooses which instance to use when resolving 
references to these routines. 

Trees, lists, and other pointer-based data structures are difficult and sometimes imprac- 
tical to implement in distributed computing models without a shared address space. The 
SUN Remote Procedure Call dereferences pointers to pass individual elements of a pointer- 
based structure [16,17]. Pointer dereferencing is adequate for situations where single struc- 
tures are passed by pointer instead of value. Others have advocated the use of subroutines to 
encapsulate access to pointer-based structures [46,66]. This approach implies changing (or 
deliberately designing) applications to encapsulate accesses to these structures. The CLASP 
software architecture solves this problem by ensuring that the context for a pointer (i.e., its 
address space) can be transferred to the remote processor. Applications may use pointers as 
handles to objects and for true pointer-based structures without concern about where a pro- 
cedure is implemented. 

CLASP uses demand paging to move arguments and data to the server. As an example, 
binary searches through large sorted arrays can be efficient because the accessed portions of 
the array are transferred to the remote processor on demand instead of prepaging the entire 
array to the server. Pages, once transferred to the server, remain on the server until they are 
required by the client processor. Pages used only by the client remain on the client; pages 
used only by the server will be transferred to and remain on the server. Pages of data used 
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by both processors migrate between hosts on demand. 

Although CLASP appears to be an approach to distributed computing, it is actually an 
extension of the traditional single-processor model onto a new underlying implementation 
that provides improved performance. CLASP mimics this single processor model but allows 
the most appropriate CPU to execute appropriate parts of the problem. It does not require 
restructuring of applications. Only portions that execute on a remote processor need to be 
recompiled. The choice of which processor performs a specific routine affects only the pro- 
cessing rate for that procedure. The choice does not alter the semantics for that procedure 
nor its interactions with other procedures in the address space. 


1.2. Thesis Organisation 

Chapter 2 describes some of the work that motivated this thesis. Chapter 3 provides a 
formal definition of the components of the CLASP system. Chapter 4 describes our proto- 
type CLASP system and the protocols it uses to communicate between processors. Chapter 5 
presents performance figures for our CLASP prototype. It also points out factors to consider 
when partitioning an application to use CAPCs. The final chapter summarizes our results 
and considers some additional research based on the CAPC concept. 
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CHAPTER 2. 
RELATED WORK 


This chapter presents a summary of other research directed towards sharing resources 
and providing language level support for this sharing. Two primary areas are explored: 
language features that provide access to other processors and system designs that support 
shared resources. We do not discuss language constructs that support concurrent processes. 
Although concurrency provides a foundation to reduce the execution time of an application, it 
usually requires that programmers manually restructure source code to use different algo- 
rithms. In this thesis, we direct our efforts towards making a single control thread execute 
faster. 

The first portion of this chapter concentrates on language mechanisms. The approaches 
discussed in these sections represent different mechanisms for transferring a control thread 
between processors. 

The next portion of this chapter discusses network filesystems. Network filesystems 
remove restrictions on where applications can execute by making the data required for those 
applications available from almost any processor. This flexibility encourages a migration 
towards workstations that provide effective work environments. 

After the network filesystem discussion, this chapter presents several distributed sys- 
tems. Two types of distributed systems are discussed. The first class extends the operating 
system to include many component systems. The second class of distributed operating sys- 
tem moves traditional operating system services, like filesystems, out of the kernel and into 
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application programs. 

We discuss multiprocessor systems in the penultimate section of the chapter. Many of 
these systems are suitable as the compute servers that we want to use for the compute- 
intensive portions of our applications. Some of these systems use special compilers to convert 
sequential applications to concurrent applications. This allows applications to use all of the 
processors in these systems and reduce the computation time. 

The chapter closes with a summary of these research efforts. We show how these sys- 
tems do not meet all of the criteria presented in chapter 1. 

2.1. Language-based Partitioning Mechanisms 

In this section, we describe several language-based partitioning mechanisms. These 
mechanisms allow applications to perform computations on remote processors. Each of these 
techniques imposes some restrictions on the applications program. Some require applications 
to be recoded in a new language. Others restrict the operations and data types that can be 
used in remote operations. 

The Remote Procedure Call uses the subroutine call abstraction as a logical point to 
transfer control between processors. The Interface Compilers described improve the imple- 
mentation of Remote Procedure Call systems. 

Distributed Path Pascal provides access to remote objects within the Path Pascal 
languages. Although the source code must be changed to use remote objects, the changes are 
minor and do not affect the existing interface to an object. Object-oriented systems provide 
support for remote operations. Eden, Smalltalk, and other object-oriented systems provide 
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support for objects that reside on other systems. However, these object-oriented systems and 
Distributed Path Pascal require that an application be coded in the appropriate language to 
use these features. 

2.1.1. The Remote Procedure Call 

Remote Procedure Calls — also called RPC — build on the control transfer intrinsic to 
a procedure call and extend this control transfer across machine boundaries. 1 Nelson argues 
that RPC is a satisfactory and efficient programming language primitive for constructing dis- 
tributed systems [66] . The RPC model consists of a client process that invokes subroutines 
implemented by a server process. The client and server are separate processes and execute in 
their own address space. Servers advertise a set of subroutines that clients can invoke. 

Because the client and server do not share a common address space, RPC clients and 
servers can only communicate through the parameter lists and return values of the subrou- 
tines advertised by the server. 2 Client and server procedures can not pass information 
through global variables because the two processors do not share an address space. In his 
thesis, Nelson suggests that procedural interfaces be used for access to global variables 
[46,66]. In a general RPC system, a server can call a routine on the client to retrieve a global 
variable. By encapsulating access to global variables, programs can be partitioned (and 
repartitioned) across clients and servers at later times with less chance of error in routines 

1 An RPC call does not have to go to another machine. The RPC server can be located on the same CPU, but 
within a different process. 

2 We discount the possibility that the client and server exchange information through a shared filesystem. This 
approach suffers from the same limitations: the client and server must take explicit action to transfer informa- 
tion between each other. Although filesystem communications might provide for more information to be passed 
in a single transaction, it is not transparent. 
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that depend on access to global variables. 

Some RPC implementations require special calling sequences to invoke routines on the 
server [ 17 , 19 ]- Figures 2.1 and 2.2 depict the client and server code segments used to invoke 
a remote procedure using the Sun RPC implementation. 

RPC implementations often work across heterogeneous hardware. The client and pro- 
cessor may have different instruction sets, processing speeds, and data representations. To 
accommodate the different data representations, arguments and results of remote procedures 
are coerced to a standard representation before being sent to the peer process. When 
received, these values are again coerced, this time from the standard order to the order used 
by the receiving processor [8,13]. This makes the RPC mechanism available across a diverse 
combinations of processors. To implement an RPC system, a process must be able to coerce 
data between its internal representation and the network standard representation. 

The generality of a standard network representation introduces several costs to a 
program’s execution. Systems must always convert data to the standard representation 
before sending it across the network; the recipient must always convert from the standard 
representation to its internal representation. 3 If the client and server share a data representa- 
tion that differs from the network standard order, RPC subroutines convert the data twice 
when it could have been passed without change. A number of processors use the Network 
Standard Order for their internal representation [4-7,47,48]. These machines can send data 
across the network without any format conversions. However, RPC calls marshall their 
parameters or results into a single buffer as part of the conversion procedure. Often, this 


* The recipient must perform this conversion. Intermediate sites, providing gateway functions, pass the data 
without conversion. 



/* 

* result = foo(5, 7, ’c*. "a string"); 

*/ 

struct foo_arglist 

< 

long argl ; 

long arg2; 

char arg3 ; 

char *arg4; 


caller () 

long result; 

int failed; 

struct foo_arglist fooargs; 

fooargs.argl = 5; 
fooargs. arg2 = 7; 
fooargs . arg3 = ’ c ’ ; 
fooargs. arg4 = "a string"; 

failed = callrpc (HOST, PROGRAM, VERSION, PROCEDURENUMBER , 
xdr_fooargs, ftfooargs, xdr_long, iresult) ; 
if (failed) 
exit (1) ; 

/* 

* "result" contains return value from foo. 

*/ 


xdr_f ooargs (xdrs , fp) 
register XDR *xdrs ; 
struct fooargs *fp; 

< 

return (xdr_long (xdrs, &fp->argl) && xdr_long (xdrs, &fp->arg2) && 
xdr_char (xdrs, &fp->arg3) && xdr_string (xdrs, &fp->arg4)); 

> 


Figure 2.1 

Sample SUN RPC Client Code 
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foohandler (rqstp, transp) 
struct svc_req *rqstp; 

SVCXPRT * transp; 

< 

long value ; 

struct foo_argllst fooargs; 

If ( ! svc_getargs (transp, xdr_fooargs, ftfooargs)) 

< 

fprlntf (stderr, "unable to decode arguments\n") ; 
exit (1) ; 

> 

value = foo (fooargs .argl, fooargs. arg2, fooargs. arg3, fooargs . arg4) ; 
if ( ! svc_sendreply (transp, xdr_long, fcvalue)) 

< 

fprlntf (stderr, "cant reply to caller\n") ; 
exit (1) ; 

> 

svc_freeargs (transp, xdr_fooargs, fefooargs) ; 
return; 

> 

long foo (argl, arg2, arg3, arg4) 

long argl ; 

long arg2 ; 

char arg3 ; 

char *arg4 ; 

{ 

/* foo procedure Implemented here */ 

> 

main () 

< 

reglsterrpc (PROGRAM, VERSION, PROCEDURENUMBER , 
foohandler, xdr_fooargs, xdr_long) ; 
svc_run () ; 

prlntf ("returned from svc_run — bad news\n") ; 
exit (1) ; 

> 


Figure 2.2 

Sample SUN RPC Server Code 
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copy operation is performed even though no format conversion is done. 

Different data types require different conversions between internal and network 
representations. To provide the correct mappings, the RPC systems must provide type infor- 
mation for procedure arguments. Constructs that abuse type information can fail in an RPC 
environment. As an example, the C cast operation allows programs to interpret the same bit 
string several different ways. 

The RPC client and server have separate address spaces. This separation provides an 
easy way to replace the code of a remote procedure dynamically. New RPC client requests 
can be sent to the new implementation of the server. RPC servers can be removed as their 
clients finish execution. 

Because RPC clients and server exist in separate address spaces, clients are not bound to 
specific servers until runtime. Different instances of the client may interact with different 
server programs. This can be used to reconfigure programs at runtime with a minimum of 
effort. In an RPC-based GKS implementation, the client process chooses a server appropri- 
ate for the desired output device [76]. The X window package uses the deferred client/server 
binding in the same manner [43-45]. 

Some RPC systems do not provide mechanisms for routines on the server to invoke 
arbitrary functions on the server [2,3,16]. The relationship between the client and server is a 
strict master/slave relationship. 
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2.1.2. Interface Compilers 

The set of procedures, their arguments, and their results make up a protocol between 
the RPC client and server . 4 To ensure that both the client and server obey the same protocol, 
a number of interface compilers have been developed. 

Interface compilers accept definitions of a procedure’s arguments and results. They out- 
put two code segments — one for the client and one for the server. The client code segment 
is a set of routines that resembles a local instance of the subroutine. This client routine pack- 
ages its arguments, transmits an RPC request to the server, retrieves the results and unpacks 
them. The server code segment unpacks an RPC request and invokes the true subroutine, 
which is implemented on the server. 

Interface compilers reduce the complexity of managing RPC interfaces. Courier, rpcgen, 
and related compilers take a specification of the input and output parameters for a procedure 
and generate the necessary subroutines to package the arguments, transmit them to a server 
processor, unpack the arguments, execute the routine on the server, and return the results to 
the client processor. 

Interface compilers do not change the semantics of the remote procedure call. They 
simplify the specification, implementation, and management of the set of routines that an 
RPC client can invoke on on an RPC server. In the next two sections, we discuss two inter- 
face compilers: Xerox Courier and Sun RPCL. Both compilers allow specification of the 
available procedures and the arguments and results for those procedures. Both compilers 
generate the data conversion and packaging routines to translate between internal 


4 Protocols often are associated -with message passing. See [59] for a discussion of how the Remote Procedure Call 
and message passing are duals. Message passing can model RPC and RPC can model message passing. 
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representation and the network standard order. The Courier system also generates local stub 
routines that can be called using local procedure call mechanisms. These stub routines per- 
form the actual RPC call. 


2. 1.2.1. Xerox Courier 

Xerox’s Courier language provides a way to specify procedure interfaces. The Courier 
compiler translates the simple specification into the appropriate client and server stub rou- 
tines. Courier-generated stub routines resolve data representation differences between hosts 
by converting parameters and results to a standard order. Simple data types are converted 
to a standard ordering for transmission across a network. Complex data types, such as 
records, are decomposed until they are a collection of simple data types. The pieces of a com- 
plex data type are reassembled on the remote host, using that host’s alignment and data 
representation. If the Courier specification is correct, the resulting stub routines will be 
correct. 

Figure 2.3 contains a Courier specification for the subroutine foo. This concise 
definition can be compared to the more complex notation in figures 2.1 and 2.2 for Sun RPC. 
With Courier, a client invokes the local stub routine for foo. This stub routine packages the 
arguments and sends them to the remote processor. 
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Foo: PROGRAM = BEGIN 

— Foo entry point 

foo : PROCEDURE [argl : INTEGER, arg2 : INTEGER, 

arg3 : CARDINAL, arg4 : ARRAY 32 OF CARDINAL ] 
RETURNS [ result : INTEGER ] 

= 0 ; 


END. 


Figure 2.3 

Sample Courier code 


Xerox has developed other RPC stub compilers. The Lupine compiler generates RPC 
stubs for the Cedar language. Versions of the Courier and Lupine compilers generate stubs 
for languages like Mesa, Interlisp, C, Smalltalk, and others [26] . 


2.I.2.2. SunRPCL 

Sun has developed an interface compiler for their Remote Procedure Call Language, 
RPCL [16,19]. RPCL uses a C-like syntax to specify the datatypes, procedures, and versions 
of an RPC server. The RPCL compiler, rpcgen, produces header files for inclusion in applica- 
tions code, generates the XDR routines for converting data to the external data representa- 
tion, and produces a server program to register the program. RPCL permits one argument 
for each remote procedure. RPCL does not build stub routines that allow users to invoke 
remote routines with multiple arguments like: 

foo (1, 2, V, "a string”); 

Instead, the argument must be collected into a single structure to be passed to the server. 
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Figure 2.4 shows the RPCL specification for the foo subroutine depicted in figures 2.1 
through 2.3. RPCL automates the generation of the packaging and conversion routines and 
structures (the structure foo_arglist and the routine xdrJooargsQ in figure 2.1). It does 
not remove the extra work in the calltri) routine in figure 2.1. 

The single argument restriction becomes important when the local or remote routines’ 
calling conventions are not under the control of the programmer configuring them for an 
RPC environment. In such cases, the programmer can not change the procedure interface so 
that it accepts a single complex argument. Instead, he must manually generate an extra 
layer of interface routines to convert between an expanded argument list and a single struc- 
ture. The Courier RPC language provides these stubs; RPCL does not. 


struct foo_argllst 

{ 

long argl ; 

long arg2; 

char arg3 ; 

string arg4 [] ; 


program RBEPROGRAM 

< 

version RBEVERSION 

< 

long foohandler (foo_argllst) = 1; 
> = 1 ; 

> = 100000 ; 

Figure 2.4 

Sample SUN RPCL Code 
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2.1.3. Network Data Segment 

The network data segment, or NDS, is an extension of the remote procedure call that 
allows the client and server to share access to global variables [56]. Like RPC, an NDS task 
comprises two processes — one on the client system and one on the server system. NDS adds 
a data segment that both client and server can access. This allows the client and server to 
share access to global variables. These processes share a specific range of their address space. 
Each processor can see changes made to data in this address range by the other processor. 
The two processors do not share physical memory; instead, they use networking hardware — 
like Ethernet — to transfer pages between themselves. The NDS software architecture is tar- 
geted for languages like FORTRAN, where the client and server routines both access vari- 
ables in COMMON. 

The virtual memory subsystem and networking capabilities of the host systems provide 
the means to share parts of the address space between the processors. The shared data 
resides at the same addresses in each process’ virtual memory address space. Figure 2.5 
shows the address space layout for the NDS architecture. The NDS software keeps one copy 
of each page in this range. These pages are demand paged between the processors as needed. 
When the client accesses a page, the page migrates to the client’s physical memory. When 
the server makes a reference to that page, it generates a page fault. The pagefault software 
retrieves the non-resident page from the client processor instead of the local backing store. 
Shared pages of the address space stay resident on the processor that last accessed them. In 
long running programs, shared pages eventually will reside on the processor where they are 
accessed. Pages accessed by both processors will move between the processors as needed. 
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The NDS software architecture supports languages like FORTRAN, where all variables 
have static addresses. NDS does not work as well with stack based languages, such as C, 
where many variables are stored on the stack. The NDS client and server processors each 
have their own stack; neither processor can access data stored in the other processor’s stack. 
Thus, the NDS architecture does not provide complete support for languages that store vari- 
ables on the stack. 
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Figure 2.5 

NDS Address Space Layout 


To execute an NDS program, the user must have two programs — an executable for the 
client and an executable for the server. The code for each processor is stored in a separate 
executable file. Special NDS compilers require information about routines are implemented 
locally and remotely. For remote routines, the NDS compiler generates RPC stubs to invoke 
the routine on the remote processor. Because the client and server programs are the result of 
several different compiler runs, the compilers for both processors must guarantee the same 
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ordering, alignment, and length of operands. Client and server compilers must assign loca- 
tions with COMMON blocks in the same fashion. 

2.1.4. Distributed Path Pascal 

Distributed Path Pascal combines the parallelism features of a concurrent language with 
access to remote resources [55]. Kolstad’s thesis describes an enhancement to Path Pascal 
that provides remote objects. These remote objects may reside on other machines. Refer- 
ences to entry procedures of a remote object are processed in an RPC fashion. Distributed 
Path Pascal packages the arguments and sends a message to a server on the remote host. 
The server then unpacks the arguments and invokes the object’s entry procedure. When the 
procedure terminates, control returns to the calling process. 

Distributed Path Pascal provides a language-level mechanism for access to remote 
resources. It does, unfortunately, require changes to the source code to use the remote opera- 
tions. These changes are restricted to a change in the declaration of the object. 

Another shortcoming in the Distributed Path Pascal approach is that all operations on a 
single object occur on the same processor. If an object encapsulates a large database, all 
operations on the database occur on the same processor. Distributed Path Pascal does not 
allow a fast lookup operation to be implemented on the workstation and an expensive re- 
ordering of the database to be implemented on a compute server. 

2.1.5. Object-Oriented Systems 

Some object-oriented systems provide mechanisms to access objects on remote hosts. 
Many of these systems uses messages as a communications mechanism between objects. 
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Messages to remote objects require extra processing at some level to move the message across 
a communications link. Remote objects do not change the nature of the language. 

These systems require different programming techniques. For example, large FOR- 
TRAN codes do not port directly to these systems. 

For example, the Eden system provides location-independent names for individual 
objects in the system [23]. Each Eden object, or eject, has its own address space. Objects 
can move between processors with the same architecture. An eject defining a matrix might 
define functions to invert the matrix, lookup specific elements, and replace specific elements. 
All of these operations are implemented on the same architecture. 

2.2. Networked Filesystems 

Network Filesystems direct their efforts towards sharing data among systems. While 
these efforts are less ambitious than a complete distributed system, they provide a useful level 
of sharing — particularly in UNIX-like systems that rely on the filesystem for most commun- 
ications between processes. Network Filesystems provide a framework that can be extended 
to build a distributed system [34]. 

Network filesystems are useful because they allow users to work on arbitrary machines 
— instead of being forced to work on the machine that holds their data. If a user can access 
his supercomputer files from both the supercomputer and a workstation, he will often choose 
the workstation for its superior work environment. With network filesystems (that allow 
users to access supercomputer files from workstations) and the results of this thesis (which 
allows users to access supercomputer cycles from workstations), users can exploit both sys- 
tems easily. 
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There are many designs for, and implementations of, network filesystems. These 
include: Xerox Alpine, SUN ND, SUN NFS, and IBM RVD [18,20,26,85]. Distributed sys- 
tems like LOCUS and the Newcastle Connection provide network filesystems as part of their 
larger goals. 

2.2.1. Xerox Networked Filesystems 

Researchers at Xerox PARC have developed several distributed filesystems. These 
include Juniper, the Interim File Server, and the Alpine filesystem. 

The Juniper filesystem, also known as the Xerox Distributed Filesystem, is an effort to 
support access to shared databases in the Xerox environment — an environment where all 
shared files are stored on file servers [65]. Because the XDFS filesystem is targeted for shared 
database systems, it had to provide support for common database operations. XDFS pro- 
vides random access files and atomic transactions. Juniper was implemented on the Alto, a 
16 bit workstation. After experimenting with this implementation, it was discovered that the 
performance was slow (but tolerable) and that the server frequently crashed. Server recovery 
took over an hour. In addition, new software systems being developed at Xerox provided a 
new basis for a more efficient and more robust file server [26,80]. This new filesystem is the 
Alpine filesystem. 

Another file server in use at Xerox is the IFS or Interim File Server. IFS differs from a 
true filesystem in that it moves the entire contents of a file to the local processor, allows it to 
be modified, and replaces the copy on the file server at the end of a session. Because IFS does 
not support random access files, it was not considered a candidate for extensions to support 
database applications. 
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The Xerox Cedax environment uses the Alpine filesystem. Alpine’s primary purpose is 
to store files that represent databases [26]. It also provides support for ordinary files contain- 
ing documents and programs. Alpine uses lessons learned from the design and implementa- 
tion of the Juniper filesystem. Alpine clients use Cedar’s RPC support mechanisms to com- 
municate with an Alpine server. The directory mechanism, providing mappings from user 
supplied filenames to the internal Alpine names, is not part of the Alpine filesystem. Instead, 
this naming system is itself an Alpine client. 

2.2.2. Apollo Domain 

The Apollo Domain operating system, Aegis, is a networked operating system that pro- 
vides transparent access to remote files and devices [9,10]. Files and devices are shared across 
the nodes of a Domain system. Each file or device has a unique internal identifier that 
describes its location in a network of Apollo systems. Directory services map external names 
into one of these unique identifiers. Aegis’ demand-paging support determines the proper 
location and arbitrates access to these resources. 

2.2.3. Sun Remote Filesystems 

Sun provides two (sometimes confused) disk-related network protocols. The first of 
these, the ND protocol, provides block-level access to remote disks. It does not implement 
remote filesystems. The more recent NFS network filesystem uses a finer granularity to pro- 
vide read/write sharing of the same filesystem by several clients. 
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2.2.3.I. The Sun ND Protocol 

The ND (Network Disk) protocol provides diskless workstations with access to disks on 
remote systems. The ND protocol provides a block level access to the raw device; the filesys- 
tem is not part of this protocol. It provides the client with a device driver that maps disk 
I/O requests into network messages. ND servers interpret these messages as requests to read 
and write specific sections of their local disks and transmit the results to the ND client. To 
the ND client, an ND disk is similar to a local disk. This ND device has a different set of pro- 
cedures to transfer data; instead of manipulating registers on a controller to start data 
transfers, the device driver builds a network packet and sends it across an ethernet. 

The ND server allocates sections of the local disk to individual ND clients. The lower 
per-byte costs and better performance of large disk drives can be shared by several worksta- 
tions. While the ND protocols allow several workstations to share the physical disk drive, 
the data on the disk drive is not shared. Each client has exclusive read/write access to a por- 
tion of the physical disk drive. Clients can share portions of the drive in read-only mode. 

Diskless workstations can use the ND device driver model to boot from remote disks. 
Some other network filesystems require local disks to boot individual processors. These disks 
usually provide little storage space and relatively slow transfer rates. 


2.2.3.2. The Sun NFS Network Filesystem 

The SUN Network File System extends the UNIX Filesystem onto a network [15,85]. 
NFS servers export filesystems; NFS clients mount these filesystems. A server allows many 
clients to access the same filesystem. Clients can perform read and write operations on the 
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filesystem; the NFS protocols keep the filesystem in a consistent state. 

The NFS design uses stateless servers. State information, like file offsets and inode 
information, is sent to the client after each operation. The client presents this information 
for later transactions. This approach has several advantages. Because the client holds the 
state information, servers can crash and reboot without disrupting service (although clients 
are delayed until the server recovers). Thus, the NFS protocols are robust across certain 
failure modes. Because servers maintain no state information, they can handle arbitrary 
numbers of clients — the server does not keep any per-client tables. Servers can handle 
many clients with low traffic levels as well as a small number of clients generating high traffic 
levels. 

To guarantee that write operations have completed, NFS servers use synchronous I/O 
to the local disk. When writing to an NFS server, the client blocks until the transfer is com- 
plete. This has a significant negative affect on performance. Some NFS implementations 
allow asynchronous writes in their servers [86]. This change can more than double writing 
throughput for applications that transfer large amounts of data. It also means that some 
types of server failures leave files partially updated and provides no indication of this to the 
client. 

The NFS protocols do not provide file locking nor do they guarantee that each write is 
atomic. File locking requires that the server maintain state information. Additional proto- 
cols, in parallel with NFS, do provide file locking primitives on NFS partitions [82]. 

Large write operations on a standard UNIX filesystem are a single, atomic operation. 
Large writes may require several network transactions between the NFS client and NFS 
server. Because the server does not maintain state, the single large write operation is broken 
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into several consecutive operations. It is possible for several clients to interleave their 
requests in such a way that the final contents of the file are a combination of data from the 
clients. 

2.2.4. IBM Remote Virtual Disks 

The IBM Remote Virtual Disk protocol provides block-level access to remote disks. 
This protocol provides functions similar to those provided by the Sun ND protocol. It has 
some additional functions that manage protections on disk partitions [18]. 

2.3. The Distributed System Model 

Distributed Systems combine networks of machines into what gives the appearance of a 
single system. They usually implement some form of network filesystem; files and devices 
generally are available from any processor in the system. The same mechanisms provide 
access to these resources from all processors, giving the system a measure of location indepen- 
dence. Location independence implies implicit access to remote resources; users need not use 
different constructs to access resources connected to a non-local processor. These systems are 
characterized by the following features: 
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• Files, devices, and (to a lesser extent) processors are accessible regardless of 
their location. 

• Applications are bound to a processor at execution startup. In some cases, an 
application can be moved among similar processors. 

• A distributed system can survive the failure of one or more component systems. 
When these components fail, portions of the system become inaccessible. Ac- 
cess to the remaining portions of the system continues uninterrupted. 

• Distributed systems can be augmented with additional component systems. 
This increases the total aggregate computing bandwidth of the system and al- 
lows the system to support more simultaneous users. 


The first system discussed, LOCUS, combines its many component processors to provide 
the image of a single centralized system [72,84]. The components of a LOCUS system share 
the same view of the resources in the system. A single name refers to the same LOCUS 
resource regardless of which processor interprets the name. 


The second system discussed, the Newcastle Connection, does not bind the individual 
systems as tightly as LOCUS [27]. Instead, it provides mechanisms that allow the application 
program to access resources on remote nodes without any syntax changes. Newcastle Con- 
nection systems have separate views of the combined filesystem; when presented to different 
machines, the same name can refer to different objects in the filesystem. 

Both of these systems provide access to remote resources. Such systems allow applica- 
tions to run on powerful processors while using resources connected to workstations. How- 
ever, the entire application executes on the supercomputer. These systems provide a coarse 
granularity of processor sharing that does not meet our criteria. 
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2.3.1. LOCUS 

LOCUS integrates several (possibly heterogeneous) computers into a single system [72]. 
It extends the familiar environment of the UNIX timesharing system into a multi-computer 
environment. It provides an environment that simplifies the development of distributed 
applications. 

LOCUS extends the UNIX environment onto a network of computers. Processes share a 
common filesystem, regardless of their assignment to CPUs. Processes communicate with the 
same mechanisms used in a UNIX system. Distributed applications can be modeled as a set 
of UNIX processes. The LOCUS operating system resolves issues of process location, network 
communications, and filesystem operations. 

LOCUS provides the same granularity of sharing as the UNIX system — that of the 
UNIX process. Processes are started on a single CPU; the system can move them to other 
CPUs of the same architecture. LOCUS does not provide for the same process to execute on 
multiple processor architectures. 

2.3.2. The Newcastle Connection 

The Newcastle Connection is a user-level implementation of UNIX United [27,37]. 
Russo’s thesis describes a kernel implementation of UNIX United [77]. UNIX United provides 
a framework for connecting the filesystems of individual UNIX systems into a larger hierar- 
chy. The roots of each UNIX system are directories in this extended tree. 

These extensions allow applications programs to reference remote files and devices 
without modification. To write to a remote tape drive, the user might use the pathname 
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/.. /unixb/ dev/rmtO. The leading /../indicates to the pathname resolution code that the file 
is in the root’s parent directory, then down into the unixb directory. The unixb directory is 
actually the root directory of another UNIX system named unixb. The UNIX directory struc- 
ture can be used to group the systems of various departments into appropriate groups. A 
user in the Electrical Engineering Department at the University of Illinois might access the 
password file on a machine in the CS department with the pathname /../../cs/a/etc/passwd. 
To access the password file on a machine within the CS department at Purdue, he might use 
/../../../purdue/cs/mordred/etc/passwd. This scheme provides an infinitely extensible nam- 
ing tree above the local roots of each system. However, Newcastle Connection names are not 
location independent. The user must have knowledge about the meta-structure of the tree 
combining systems and knowledge about current node’s location in this meta-structure. 

In addition to allowing access files and devices on remote systems, UNIX United pro- 
vides for program execution on remote processors. This can allow for faster execution of pro- 
grams by specifying that they run on faster or less loaded CPUs. The mechanism for specify- 
ing the CPU to execute a program is tied to the system where that binary resides. If a binary 
exists on system A, it executes on system A. The UNIX pipe construct can be used to build 
series of connected programs and execute them in parallel on separate processors. The UNIX 
text processing stream makes a good example of this feature: 

tbl J eqn J pic J ditroff j d300 

By specifying program images on separate hosts, the separate processes can be scheduled 
on 5 different processors. This provides the potential for a five-fold throughput increase; the 
actual improvement is somewhat less due to the communications costs between processors 
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and the synchronization that occurs at the original hosts [37,77]. 

Like LOCUS, the Newcastle Connection extends the scope of a UNIX process. Processes 
can now access remote resources. However, CPU sharing still occurs at process granularity. 
Individual processes can not execute on multiple CPU architectures. 

2.4. The Client/Server Model 

The client/server system is another form of distributed operating system. These sys- 
tems use a different approach to provide services traditionally implemented by the operating 
system kernel. Client/server systems demonstrate a trend toward reducing the size of the 
operating system; the major function of these operating system kernels is to provide message 
passing between processes. Server programs provide the services provided by the kernel of 
more traditional operating systems. To the (new) kernel, these server programs are addi- 
tional user applications which might be located on non-local nodes. Processes wishing to use 
services such as filesystems are clients of these servers [22,30,41,74,75]. 

A result of this approach is that much of the operating system’s overhead can be moved 
to another processor. In some cases, the operating system overhead can consume 50 percent 
or more of the CPU cycles. When most of the operating system overhead is removed, appli- 
cation programs can achieve higher sequential processing speeds on that CPU. Communica- 
tions between the application and the traditional operating system services are now more 
expensive: the new arrangement introduces communications costs between the application 
and the operating system. 

Client/server systems are often implemented in environments with many processors. 
To exploit the parallelism available across these processors, an application program must be 
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partitioned manually into concurrent processes. Client/server systems concentrate on pro- 
viding efficient communications mechanisms between processes and processors because the 
system performance is so dependent on this underlying communications mechanism. Slow 
communications mechanisms can reduce the system throughput dramatically. 

The subsections below describe several systems based on the client/server model. Each 
provides the programmer with the ability to partition application’s into separate pieces to 
execute in parallel on separate processors. All of these systeihs can execute several instruc- 
tion streams from the same logical address space. In many cases, the degree of concurrency is 
limited to the number of processors sharing physical memory. This allows reduced solution 
time for some appropriate algorithms in systems configured with multiple processors sharing 
a single memory. 

2.4.1. Amoeba 

The Amoeba operating system is a client/server system being developed at the Vrije 
University in Amsterdam [81]. The Amoeba system comprises four component types: 

• Workstations, one for each user, 

• Pools of processors, dynamically assigned to user tasks, 

• Specialized servers (e.g., file servers, bank servers), and 

• Gateways which link Amoeba systems at separate sites. 

The Amoeba kernel provides message passing facilities and few other functions. This is 
motivated by the desire to keep the kernel small, enhance its reliability, and provide the trad- 
itional kernel features in user processes to facilitate flexibility for experimentation. 

Amoeba assigns tasks to processors at process creation time. It is possible for several 
processes to share the same text and data segments though the individual processes each have 
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private stack segments. Processes which share text and data share the same processor. 
Because applications can dynamically acquire and release processors, the aggregate 
bandwidth available to an application can be quite large. Applications must still be explicitly 
partitioned properly to take advantage of this bandwidth. 


2.4.2. Stanford V System 

The V kernel project at Stanford University provides services in an environment of disk- 
less workstations connected by a high-speed local network [30]. The V kernel provides a fast, 
message-based communications framework for RPC among clients and servers. Teams of 
one or more processes sharing a single address space provide processing power for a single 
job. 

V System message passing is synchronous. After sending a message, the sending process 
blocks until the recipient replies. Since the V kernel guarantees delivery of messages, pro- 
grammers need not implement protocols to ensure reliable communications between 
processes. Within a team, starting a new process is relatively inexpensive. These inexpensive 
processes provide a means to implement asynchronous communications; many programmers 
create inexpensive processes only to send a message, wait for the acknowledgement, and then 
terminate. 

Each V process resides on a single logical host; all processes in a V team live on the 
same logical host. A logical host resides on exactly one physical host though a physical host 
may support many logical hosts. 
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Logical hosts can be moved between physical hosts, allowing a set of processes to be 
moved from a busy processor to an idle processor; the busy processor suspends the processes 
on the logical host, creates a logical host on the target processor, and sends the state of the 
logical host to the remote host. The actual implementation allows the logical host to execute 
while its state is being shipped to the new host. After the bulk of the state is transferred, the 
V kernel checks to see what state has changed, and sends updated information for those parts 
of the logical host [83]. 

Adding more processors to a V system can improve aggregate throughput. Decreasing 
the solution time for a single application requires explicit restructuring of the application or 
using faster processors. Cheriton has developed a master-slave approach to structuring 
applications to execute on networks of personal workstations that do not share memory and 
have limited communications bandwidth [29]. 

2.4.3. CMUMACH 

The CMU MACH system provides a new foundation for future UNIX development 
[22,74]. MACH draws heavily on previous experience with Accent and uses a similar underly- 
ing model [75]. MACH is designed to provide the facilities needed to exploit general-purpose, 
shared-memory multiprocessors. 

The MACH address space (task) can have many active instruction streams (threads). 
There can be many such tasks in a running MACH system. MACH’s multiple-thread model 
provides a foundation for the easy use of parallel algorithms; shared-memory provides inex- 
pensive data transfer between the threads of a task. On multiprocessors, MACH schedules 
the threads concurrently and the application runs faster. On uni-processors, MACH 
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schedules the threads sequentially. Although execution on a uni-processor takes longer, the 
program produces the same results. 

MACH provides programmers with constructs to partition a program into parallel 
threads of computation that share a common data space. MACH also implements parallelism 
without a shared data space; such a case is a simple RPC implementation using the MACH 
message passing facilities between tasks. 

MACH provides a good base for a CLASP implementation. The new virtual memory 
management system allows non-kernel tasks to be specified as the paging mechanisms for 
individual tasks [22]. This allows processes to page to user filesystems; MACH does not 
require disk partitions to be dedicated to swapping [22]. The user level routines for CLASP 
should be easily ported to the MACH environment. The page-management routines for 
CLASP can be implemented using a non-kernel task that is assigned paging responsibilities 
for a process. 

2.5. The Multiprocessor System Model 

Multiprocessor systems are another way to share resources. In these systems, a number 
of processors are connected to a common memory. These processors can share peripherals. 
An multiprocessor system with N CPUs can usually provide almost N times the performance 
of a single processor system. Because the CPUs share the same set of peripherals, an N pro- 
cessor system can cost much less than N times the cost of a single processor system. 

Multiprocessor systems provide increased aggregate throughput. Because they share 
memory, these systems can balance the load on each CPU without incurring high costs to 
move jobs across a network between systems. With the appropriate compiler and operating 
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system support, these systems can reduce the execution time for applications. 

The first set of systems described below uses many processors to improve aggregate 
throughput. In these systems, the operating system allocates jobs to idle processors from a 
single ready queue. Jobs may execute on different processors from one timeslice to the next. 
Because moving jobs between processors does not incur any communications costs, a single 
multiprocessor system with N processors provides better aggregate throughput than a net- 
work of N uniprocessor systems (which have overhead when moving tasks between systems). 
Idle processors can execute any ready job because all jobs are in a shared memory. By them- 
selves, these systems do not reduce the solution time for a single application. However, appli- 
cations can be restructured to exploit the concurrency available in these systems and reduce 
their execution time. 

The second set of multiprocessor systems uses compiler technology to detect implicit 
concurrency in sequential code and constructs suitable for vector operations. Compilers for 
these systems break sequential code into blocks that can be scheduled to execute concurrently 
by the operating system. Data dependencies between blocks of code determine when blocks 
can be scheduled. These dependencies are similar to Petri nets; when the input values are 
ready, the block ^ires[7l]. These systems often use hardware technology similar to the first 
set of multiprocessors. This hardware reduces memory contention and saturation of shared 
resources through complex and expensive memory hierarchies. Some of these systems reduce 
contention for the global memory by providing each processor with a private memory. 

The multiprocessor systems described in this section comprise single logically and physi- 
cally integrated systems; failure of critical elements in these systems stops all operations. 
They are like centralized timesharing systems in this respect, although they might offer 
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higher performance or easier expansion routes than a centralized, single-processor timeshar- 
ing system. These systems also do not support the enhanced interfaces (e.g., mice, bit- 
mapped displays) available on workstations. It currently is not cost effective to support these 
interfaces on these machines for many users. Multiprocessor systems are more expensive 
than single user workstations. While some multiprocessor machines can be expanded inex- 
pensively to include more processors, there is still a significant initial cost. 5 

In addition to the systems we are about to describe, other multiprocessor systems 
include the Cray-X/MP, the CDC Cyber series, and the DECsystem 20. The DEC VAXclus- 
ter is a closely-coupled system that allows multiple processors, with separate memory, to 
share access to common peripheral devices. The VMS operating system uses this hardware to 
provide load balancing and improved system reliability [57]. 

The next sections describe several multiprocessor systems in more detail. The first sec- 
tion describes the C.mmp project, an early experiment in multiprocessor systems. The 
second section describes the Sequent and Encore systems, where the processors share a single 
global memory. The third section describes the University of Illinois CEDAR project, which 
combines multiprocessor systems with compilers that automatically restructure sequential 
applications into concurrent applications. 


* A 2 CPU, 2 Mbyte Sequent Balance-8000 system costs $59,000. A 4 CPU, 4 Mbyte Sequent Balance-21000 sys- 
tem costs $139,900. The Balance-21000 has backplane slots for more processors than the Balance-8000. Addi- 
tional processors are $10,000 per card (2 processors). An Alliant FX/8 with 1 CE costs $150,000. A fully expand- 
ed FX/8 (with 8 CEs) costs $450,000. A SUN-3/52 workstation (with a local disk and cartridge tape drive) costs 
$13,900. A diskless SUN-3/50 costs $4,995. 

The Sequent prices are as of August 1986. The Alliant prices are from June 1985. The SUN prices are from fall 
1980. 
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2.5.1. C.mmp 

The C.mmp amd Cm* systems provided an early testbed for experiments involving mul- 
tiple processors connected to a common memory hierarchy. 

The C.mmp system connects 16 PDP-11 processors to a large common memory through 
a crossbar switch [88-91]. Address relocation boards in each processor map virtual addresses 
on the private bus to physical addresses in the shared memory. Thus, each processor in the 
C.mmp system could access the entire common system memory. Private, per-CPU memory 
is used for off-line maintainence and certain private operations. 

The Cm* machine contains clusters of processors. Like the C.mmp system, each proces- 
sor has a private memory. However, Cm* has no single global shared memory. Instead, each 
cluster of 5 machines is connected with a special Kmap processor that allows the CPUs to 
access the memory connected to other CPUs. Kmap processors communicate between them- 
selves to implement cross-cluster memory accesses. Each Cm* processor can access any of 
the physical memory using a uniform method, but the access costs vary depending on the 
relative locations of the processor and memory. Access times for memory in other clusters 
can be as much as 10 times slower than access of the memory directly connected to the pro- 
cessor [68,69]. 

2.5.2. Encore and Sequent 

The Encore and Sequent systems represent a family of systems that use timeslice shar- 
ing to allow a fine partitioning of the available computing cycles provided by many processors 
to many processes. In timeslice sharing systems, a job may execute on a different processor 
each timeslice. These systems use many processors attached to a common memory to 
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improve aggregate throughput. A processor can be reassigned to a different task each 
timeslice. This technique allows a pair of 4 MBPS processors to provide service for two real- 
time processes requiring 3 MIPS and a third process requiring 2 MIPS. A pair of 4 MIPS 
single-processor systems can not meet the needs of the same 3 processes. Timeslice-sharing 
systems do not reduce the solution time for a single application. The two processor system (4 
MBPS each processor) does not meet the needs of an 8 MIPS program nor does it meet the 
needs of a 5 MIPS program. 

Explicit restructuring of applications for concurrency allows timeslice sharing systems 
to reduce the solution time for a given problem. The shared-memory between processors 
improves the performance over that achieved when the same application is spread across a 
network of processors. The communications costs between separate processes on these sys- 
tems is much lower than the cost between processes on systems separated by a network. 

The common memory provided in timeslice sharing systems can be used for an efficient 
data sharing scheme. Some of these systems permit processes to share portions of the address 
space [12,14]. Other systems, such as MACH, let processes share the entire address space. In 
each of these cases, the application must still be restructured to reduce the solution time for 
that application. 

One of the advantages provided by these machines is their ability to be expanded inex- 
pensively. If more computational power is needed, additional processors can be purchased as 
single boards. The additional boards plug into the system and give an immediate perfor- 
mance improvement. 
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2.5.3. CEDAR 

The CEDAR project at the University of Illinois is building large scale multiprocessor 
supercomputers [42], To attain this goal, the project is concentrating on connecting many 
processors to a common memory hierarchy, using compilers that automatically restructure 
sequential code into segments that can be executed in parallel, and developing a control 
hierarchy to coordinate these many parallel tasks. 

The CEDAR compilers isolate small blocks of code that can be scheduled independently 
of each other [58]. The compiler decomposes loops without recurrence relations (and those 
with specific types of recurrence relations) into several smaller loops that can be scheduled to 
run on several processors concurrently. 

The underlying hardware model for the CEDAR architecture is many processors sharing 
a common hierarchical memory and each processor having a private memory. Code and data 
required to execute a block are copied to the private memory. The output values are 
returned to the common memory. Initial implementations of the CEDAR system are planned 
for 8 and 16 processors. Larger systems, with 1024 or more processors, are being designed. 

The CEDAR system reduces the solution time of a single application by partitioning the 
application into blocks that are executed concurrently. The compiling and scheduling tech- 
niques used in the CEDAR system are applicable to a range of memory hierarchies. Systems 
with a common bus and a single global memory can show reduced execution time by using 
compilers that automatically restructure the application. The CEDAR compilers show that 
automatic restructuring techniques can be used to improve execution time for sequential 
applications in a multiprocessor environment. 
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2.6. Summary of Related Work 

We have examined a number of existing tools that provide support for resource sharing. 
However, they do not meet the criteria defined in chapter 1. 

The Remote Procedure Call allows applications to be partitioned between client and 
server systems, RPC often requires changes to source code. All communications between the 
RPC client and server must be through the defined procedural interfaces. Applications that 
pass information using global variables or that use pointer-based structures can not be parti- 
tioned transparently. RPC implements a restricted subset of the procedure call abstraction. 
The NDS architecture removes some of the RPC restrictions by providing a shared data seg- 
ment and allowing client and server routines to access shared global variables. NDS does not 
support sharing of variables stored outside of the data segment (e.g., variables stored in the 
stack). Both RPC and NDS store the client and server portions of a program in separate 
files. 

Object-oriented systems support remote operations. However, these systems require 
that the application be coded using the appropriate languages. A large FORTRAN code 
would have to be recoded in the appropriate new language to take advantage of these 
features. This violates our rule that we will not require restructuring or recoding of applica- 
tions. 

The network filesystems presented in this chapter allow applications to run on any 
machine. They are no longer restricted to executing on the system that stores the 
application’s data. Because network filesystems only provide access to data on remote pro- 
cessors, they do not solve the problem of partitioning an application so that it executes fas- 
ter. Applications can be moved, in their entirety, to faster processors. But, this does not 
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satisfy our criteria that the workstation must be used for more than submitting jobs to a fast 
processor. 

Like network filesystems, distributed systems provide access to local and remote 
resources. Applications can be moved to larger, faster processors in these systems and exe- 
cute faster. These systems move entire programs to other processors. Like network filesys- 
tems, they fail the criteria that the local workstation be used for more than submitting jobs 
to a faster processor. 

The final set of systems described, the multiprocessors, can provide improved perfor- 
mance. They provide hardware support for compilers that analyze sequential programs to 
generate concurrent programs. Because of their relatively high costs, these systems are more 
appropriate for compute servers than as workstations or clients. 
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CHAPTER 3. 

CLASP AND THE CROSS ARCHITECTURE PROCEDURE CALL 


The CLASP architecture is designed to provide compute servers for workstation users. 
The compute servers are connected to the workstations using local area networks like Ether- 
net. To meet our definition of effective, these compute servers must: 

• exploit the speed difference between client and server processors, 

• require no changes to application code, 

• continue to use the workstation for appropriate portions of the computation, 

• have no impact on the use of workstation-specific interfaces and devices such as 
bit-mapped displays, mice, and windows. 

The CLASP architecture implements the traditional UNIX process model on a new 
underlying foundation. This new foundation allows appropriate sections of an application to 
execute on the most appropriate processor. Like the Remote Procedure Call and NDS archi- 
tectures, the CLASP architecture partitions applications at the procedure call level. CLASP 
differs from these architectures by providing a more transparent mechanism to access remote 
procedures. CLASP also removes some restrictions imposed by these other architectures. 
Procedure calls that cross architectural boundaries are named Cross Architecture Procedure 
Calls (or CAP (7s) in the CLASP software architecture. 

CLASP allows a set of routines within an application to be accelerated by recompiling 
them for a faster processor. They will be executed on that processor. CLASP manages the 
movement of data and the control thread between processors. CLASP seldom requires source 
code changes to exploit the speed advantages of the compute server. 
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A new CLASP loader collects object files for multiple processor architectures into a sin- 
gle executable file. The loader groups instructions for a particular processor architecture into 
a contiguous virtual address range. The resulting executable file contains information that 
describes the processor architectures required to execute different portions of the address 
space. 

A set of modifications to the operating system kernel handle the runtime details of the 
CAPC software architecture. These modifications handle the detection of accesses to code for 
non-local architectures, the transfer of control to a processor of the correct architecture, and 
the transfer of pages in the address spaces between processors. These operations are tran- 
sparent to the application program; the kernel implements all of these operations. 

An application’s performance is controlled by the way it is partitioned between the 
client and server. Some partitionings yield better performance than others. It is not efficient 
to use.CAPCs to calculate a square root; the network overhead overwhelms any speed advan- 
tages provided by the remote processor. An application is partitioned by compiling specific 
source files for specific architectures. 

This chapter describes the characteristics that the workstation and compute server must 
share. It describes the virtual address space of a CLASP process. One section describes how 
CAPC control transfers are detected and processed. Another section describes process state 
manipulations, including changes in the virtual address space and paging between systems. 
The chapter closes with a comparison of CAPCs and RPCs. 
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3.1. Required Homogeneity 

The CLASP architecture requires that processors share data representations and have a 
common address space. However, each processor may have different instruction sets, register 
sets, and stack frame formats. 


3.1.1. Homogeneous Data Representation 

CLASP requires that all participating processors agree on how data is represented. F or- 
tunately, many processors share a common data representation. For example, the Motorola 
68000 and IBM RT-PC microprocessors have the same data representation as the Convex 
C-l and Alliant FX computers [1,4,7,11,47,48]. Because they share the same internal data 
representation, these processors can exchange information without intermediate format 
conversions. The processors can move data as a series of bytes without interpreting the con- 
tents and without providing type information. 

When compiled and executed on different processors with the same data represenation, a 
subroutine generates the same output. We are not concerned with the actual machine 
instructions that implement the subroutine, but rather with the relative processing rates of 
the two processors. One processor may execute the subroutine signficantly faster than the 
other. For example, a CPU with vector instructions might execute a matrix multiplication 
subroutine significantly faster than a CPU without vector instructions. Each processor exe- 
cutes a different sequence of instructions at different rates, but the generated output is the 


same in both cases. 
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3.1.2. Homogeneous Address Space 

Many languages provide pointer data types in addition to character, integer, and float- 
ing point data types. A CLASP system must provide a homogeneous data representation for 
pointer data types. To provide support for pointer types in a CLASP system, the processors 
must share a common virtual address space. The common virtual address space allows the 
client and server processors to share pointer data types. Therefore, CLASP implementations 
for languages that provide pointer data types require that client and server processors have 
an intersecting virtual address space. An exact address space match is not necessary; a 
CLASP system can function with a sufficiently large intersection. 

Some application languages, like FORTRAN, do not provide pointer data types. For 
these languages, a common virtual address space is not necessary. Instead, a direct mapping 
between the two address spaces is sufficient. Modifications to the loader to handle the skew 
in the address space allows these languages to be split between client and server processors. 
The loader adjusts operand offsets to accommodate the different locations in the virtual 
address space. 


3.1.3. Homogeneous Compilers 

Identical data representations and intersecting address spaces provide a base for the 
CLASP software architecture. To use this system, the compilers for both architectures must 
share several properties. In particular, variable types (e.g., int, long, float, etc.) must 
translate to the same length, alignment of variables within structure or record definitions 
must be the same, the compilers must use similar argument passing techniques, return values 
must also be compatible. Many UNIX implementations include C compilers based on the 
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portable C compiler and share many of these properties [52]. 

High-level programming languages provide several basic data types (e.g., int, float, 
short, long, etc.) and mechanisms for constructing more complex data structures (e.g., the 
PASCAL record or the C struct). Both compilers must map simple data types to the same 
sizes. A compiler that translates int into 2 bytes of storage is incompatible with a compiler 
that maps int into 4 bytes of storage. For more complex data types (e.g., record or struct), 
the compilers must obey the same alignment restrictions. Some compilers align to even byte 
boundaries to meet processor restrictions. Other compilers align to 4 byte boundaries to 
match memory subsystems. A CLASP system requires that the compilers use the same align- 
ment rules. 

The CLASP routines must be able to pass a subroutine’s argument vector to the remote 
processor. In many cases, argument vectors are stored as a contiguous array of bytes. The 
first parameter is stored at the low address of the byte array and the last parameter is stored 
at the high end of the array. Other compilers place the first few arguments in registers and 
store the remaining arguments as an array of bytes [67]. For these cases, CLASP kernel rou- 
tines collects the register arguments and the additional arguments into a single vector to pass 
to the remote processor. 

The CLASP routines must also be able to collect the return values. Many compilers 
place return values in one or two registers. On the Motorola 68000, return values of 32 bits 
or fewer are passed in the register DO. 64 bit return values are passed in DO and Dl. 

Routines that return complex data types are handled in several ways. 1 Some compilers 


1 This difference prevents us from using the IBM RT with its current compilers as a client of processors such as 
the SUN, Convex, or Alliant. 
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return a pointer to an instance of the complex data type [52]. This instance is in a static 
per-routine buffer. The calling routine copies the instance to its desired location. Other 
compilers use a different approach [49,67]. The calling routine prepends an extra argument 
to the parameter list. At call time, this parameter contains a pointer to an instance of the 
complex data type. The called routine stores its results in the supplied buffer. 

CLASP can use compilers that implement either technique, but both compilers must 
implement the same mechanism. Compilers that change the argument list according to the 
return value of the routine can not be easily matched with compilers that do not. 

3.2. The CLASP Address Space 

CLASP provides each application with a single address space. The client and server 
portions of an application share this address space. Individual pages of the address space 
reside in the physical memory of one processor or another. When a processor requires a page 
that resides in the remote processor’s physical memory, the data is demand-paged across the 
network and placed in the local physical memory. The page tables on both processors are 
modified to reflect the page’s new location. 

The address space contains the instructions for both processor architectures. For each 
architecture, the CLASP loader consolidates the instructions into a single range of addresses 
in the address space. A CLASP executable file contains a table describing the instruction 
space for each architecture. The loader generates this information when it builds the execut- 
able image. The CLASP kernel uses this instruction space information and the virtual 
memory protection hardware to detect CAPC calls and returns. 
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The CLASP address space is similar to the UNIX address space. However, the single 
text segment of a UNIX process is replaced with a number of text subsegments. Each text 
subsegment corresponds to a processor architecture. The instruction space table in the exe- 
cutable file describes the boundaries and architectures of these text subsegments. Figure 3.1 
shows the layout of the multiple text segments, the data segment, and the stack segment for 
the normal UNIX process and a CLASP process. 



Figure 3.1 

CLASP Address Space Layout 


The UNIX stack segment contains the activation records for called procedures. Each of 
these activation records is in the format of the local processor. The CLASP stack segment 
contains activation records for both processor architectures. CLASP inserts its own informa- 
tion between adjacent activations records for different processor architectures. This informa- 
tion allows CLASP to handle control transfers between architectures. This is described in 
more detail in section 3.2.2, The Clasp Stack Segment, and section 3.3.2, Stack Frame Mani- 
pulation. 
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3.2.1. The CLASP Text Segment 

The text segment contains the executable code for the client and server architectures. 
The loader combines the instructions for a particular architecture into a text subsegment. 
The CLASP loader aligns the text subsegments on virtual memory page boundaries. This 
lets the CLASP kernel use the virtual memory protection mechanism to prevent a processor 
from fetching instructions from another architecture’s text subsegment. Some virtual 
memory systems provide an execute protection bit. Other virtual memory systems combine 
read and execute permission into a single protection bit. In the former, a processor may be 
allowed to read the instruction space for remote architectures. Systems with a single protec- 
tion for read and execution permissions make it impractical to provide read permission for 
the pages that are non-executable. 

Each processor maintains its own page tables for residency and protection information 
about the address space. The protection information is used to detect and process subroutine 
calls that cross CPU boundaries. The client sets its page table entries for the server architec- 
ture instruction space to disallow execution . 2 Attempts to execute instructions from those 
pages on the client CPU generate traps to the operating system kernel. The server allows 
execution of the instructions appropriate to its architecture, but protects the pages with 
client architecture instructions to prevent execution of client instructions on the server pro- 
cessor. 


* Some virtual memory systems provide read, write and execute protection bits for each page. For these systems, 
CLASP clears the execute permission bit. Other VM systems combine read and execute permissions into a single 
bit. To disallow execution in these systems, CLASP must also disallow read permission on those pages. 
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3.2.2. The CLASP Stack Segment 

Like the UNIX stack segment, the CLASP segment contains subroutine activation 
records. The CLASP kernel transparently extends the stack segment to hold new subroutine 
activation records or stack frames. The CLASP system stores activation records for both 
processor architectures on the same stack. When a CAPC causes the control thread to move 
between processors, the CLASP kernel arranges for the reipote processor to have the correct 
stack frame for its activation record. The kernel builds a CAPC packet to describe the 
transformation between the processor architectures and stores this information on the stack. 

This CAPC frame contains the argument vector location and length, the called 
procedure’s entry point, the return address for the calling procedure, and the stack bounds. 
CAPC calls and returns use this frame to exchange information. The remote processor may 
replicate some portions of the procedure call information if needed. For example, the VAX 
hardware architecture provides an argument pointer register that can be loaded with the 
location of the original arguments. The Motorola 68000 family does not provide an argument 
register, instead it expects the arguments to be just above the stack pointer at procedure 
entry. A 68000-based CLASP system copies the argument vector to set up the correct stack 
layout for a procedure call. Some hardware architectures require the CLASP kernel to repli- 
cate the procedure call information. The original copy is stored in the client’s format while 
the second copy follows the server’s stack protocol. 

The local processor’s general registers are not kept in the CAPC frame. There are 
several reasons not to store the registers. The remote processor uses its own registers, it will 
not overwrite any registers on the local processor. The register layout may differ between 
processors; the remote processor might have more registers or they may have different names 
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and semantics. The 68000 uses register A7 as a stack pointer; the IBM RT uses register R1 
as its stack pointer. 

Figure 3.2 depicts the stack segment as it looks while executing a routine on the remote 
processor. Procedures 1 and 2 execute on the client; procedures 3 and 4 execute on the 
server. 

A processor can examine variables stored in the stack segment by the peer processor. 
The variables have the same internal representation; the remote processor requires only the 
address of the variable to access, and modify, the variable. 
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CLASP Stack Layout During a CAPC 
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In a single architecture model, programs are able to destroy their activation records. 
Because the CLASP kernel stores the CAPC frame in the user’s stack segment, these faulty 
programs also can destroy the CAPC frame. 

3.3. CAPC Linkage 

This section of the thesis describes how the CLASP kernel processes CAPC calls and 
returns. To process a CAPC call, the kernel must detect the reference to a non-local pro- 
cedure, identify the argument vector’s location and length, determine the return address, and 
determine the stack boundaries. After gathering this information, the kernel builds a CAPC 
frame on the user stack and transfer control to the server. The CLASP kernel on the server 
uses the CAPC information to build a calling frame in the server architecture’s format and to 
start execution of the remote procedure. 

This section also describes the mechanisms used to implement nested CAPC calls. The 
CLASP system allows server routines to invoke routines on the client. 


3.3.1. Detecting Calls to Procedures for Other Architectures 

The CLASP kernel uses the virtual memory protection system to detect CAPC calls and 
returns. The CLASP loader generates a map describing the architecture for each of the text 
subsegments. The kernel uses this to protect the instructions for non-local architectures. 
When an application makes a procedure call to an address in one of these protected regions, 
the virtual memory system generates a fault and traps to the kernel. The kernel examines 
the faulting instruction to determine whether it is a call or return. The target address for the 
call or return is the address that generated the fault. 
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For a CAPC call, the kernel also determines the location and length of the argument 
vector. This is an architecture-dependent operation. The kernel places this information in a 
CAPC frame and stores it on the stack. CAPC frames contain: 

• a location to start execution, 

• the location of the arguments to the subroutine, 

• the count of arguments to the subroutine, 

• the stack pointer, indicating where the server can build a procedure call frame, 

• information from the server used to restore the stack frame upon completion of 
the CAPC, and 

• the arguments, if their total size is less than 64 bytes. 

The kernel sends the address of the CAPC frame and a copy of the CAPC frame to the 
remote processor. The kernel on the remote processor uses the CAPC frame to build an 
activation record for the called procedure. After building the activation record, the kernel 
starts the called procedure at its entry point. 

When the remote procedure finishes, it returns to the address stored in its activation 
record. This address is in the text segment for the client processor. The virtual memory sys- 
tem generates a protection fault when the processor attempts to fetch the instruction at the 
return address. The kernel determines that a return instruction caused the fault. The server 
loads the return values into fields in the CAPC frame. After loading the return values, the 
server passes the CAPC frame to the client processor. The client processor retrieves the 
return values from the CAPC frame, loads them into the appropriate registers for the client 
architecture, removes the CAPC frame from the stack, and resumes execution of the applica- 
tion code. 
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The stack frame after several nested CAPCs might look like the frame depicted in figure 
3.3. This stack frame is for a process with procedures PI through P5. Each procedure calls 
the next higher numbered procedure. The client processor executes procedures Pi, P2, and 
P5. Procedures P3 and P4 execute on the server architecture. 

Many programming languages provide a mechanism to pass functions as formal parame- 
ters to other routines. The UNIX qsort{Z) routine uses this mechanism to allow users to 
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specify a function that returns the ordering of two elements. CLASP allows applications to 
pass both client and server procedures as formal parameters. The procedure’s architecture 
does not change how the procedure is passed. When the routine is invoked from the called 
routine, there are no special instructions that differentiate between local and remote pro- 
cedures. Appendix C contains an example program that demonstrates this feature of the 
CLASP architecture. 

3.4. Compute Servers 

When starting a CLASP process, the operating system starts any server processes that 
might be required by the client. The CLASP kernel starts these processes during the UNIX 
exec(2) system call. If the kernel can not instantiate a server, the exec( 2) call is aborted. 
These failures are mapped onto an existing failure condition for the execQ call. Failed server 
instantiations are reported to the caller as ENOMEM errors, a message that indicates the 
system was unable to allocate swap space for the new executable file. 

For some processes, a program may never invoke the routines implemented on the 
server. In these cases, the server startup cost has been wasted. However, delayed server 
instantiation introduces new failure modes into an application program. If the kernel delays 
the instantiation of the server until the application attempts to execute code for that archi- 
tecture, applications must be prepared for a potential error on any procedure call. By estab- 
lishing servers at process creation time, we do not introduce any extra failure modes. The 
execution time saved by delaying server instantiation does not justify the programming costs 
to accommodate the new failure modes. 
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3.4.1. Selecting A Server 

The CLASP kernel selects the processor that acts as a server for CAPC applications. 
Different instances of an application can use different server processors. Many clients can 
select the same server processor. 

Our CLASP implementation stores a list of <architecture,addre$s> pairs to describe 
available servers. 4 After determining the required server architecture for a CLASP process, 
the kernel searches this table for an entry with the correct architecture. 5 At this time, the 
kernel attempts to connect to the CLASP server daemon listening at the specified address. If 
the local kernel does not receive an answer, the exec( 2) call is aborted. If the local kernel 
establishes a connection with the server process, it sends a message describing the address 
space: the text subsegments, their architectures, and their address ranges. The local kernel 
then allows the application program to begin execution. 


3.4.2. Distributed Virtual Address Space 

RPC systems package and ship entire parameter lists to the server on each call. For 
large argument lists (e.g., an array of simultaneous linear equations), this operation incurs a 
significant cost. Later operations (e.g., solving a linear system for a particular right hand 
side) require the factored array to be re-transmitted. Thus, an RPC system to solve the 
linear system Ax = b might: 


4 Note that the local processor can be the server. This aids testing, but does not improve performance. 

5 The architecture information — which architectures are required and their address ranges in the text segment — 
are stored with the header information in the executable file. 
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• Transmit the A matrix to the server, where it is factored, 

• Return the factored A matrix to the client, and 

• Transmit the factored A matrix and a b matrix to the server, to obtain a solu- 
tion. 

The CLASP architecture uses demand paging to reduce paging traffic between the client 
and server systems. The CLASP kernel sends pages to the remote processor only when the 
remote processor attempts to access that page. Pages accessed exclusively by one processor 
stay on that processor. Thus, global variables accessed by a server routine will be demand 
paged to the server when it is accessed, instead of being pre-packaged and transmitted as 
part of the procedure call. 6 

After a call to the compute server, a set of pages resides on the server. These pages 
remain on the server; the server does not send the pages back to the client until the client 
requests them. This approach follows Denning’s guidelines for working sets:[31] 

The fundamental strategy advocated here — a compromise against a lot of expensive 
memory — is to minimize page traffic. 

Our implementation provides for a single copy of each page in the virtual address space. 
For read/write pages, this approach works well. For read-only pages, this approach is 
inefficient when both processors access those pages. 

More recent work at Yale has implemented mechanisms for maintaining memory 
coherency in a loosely-coupled multiprocessor system [61]. Li’s thesis provides mechanisms 
for replicating pages accessed in read-only mode. When pages are written, the system invali- 


' By dereferencing pointers and including the underlying objct, RPC systems sometimes generate less network 
traffic than CAPC systems. For further information on this, see section 3.6 CAPCs vs RPCs. 
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dates the extra copies of that page. He outlines several protocols to ensure that only one pro- 
cessor has write permission on any page and describes how the rights for a page can be passed 
between processors. Li’s research used loosely coupled homogeneous processors to run con- 
current algorithms using a network of workstations. However, the results can be used to 
reduce network paging traffic in the CLASP system. 

3.5. Process State Manipulation 

This section of the thesis describes how CLASP alfects a process’s state. A process’s 
state includes its address space, the extent of its address space, its file descriptors, and other 
information stored in the kernel. Address space information includes the boundaries of the 
text, data, and stack segments. It also includes the contents of those segments. 


3.5.1. Address Space Bounds 

The CLASP client and server processes share the same virtual address space. To ensure 
a consistent view of this address space, the CLASP kernel must arbitrate access to pages and 
the bounds of the address space. Section 3.4.2 explains the CLASP page management 
scheme. This section describes how the CLASP kernel manages changes in the address space 
bounds. 

The stack segment grows dynamically to hold extra procedure call/return information. 
The break(2) UNIX system call extends, or reduces, the size of the data segment. The client 
and server processors must inform each other about changes in the boundaries of the address 
space. Each CLASP process must be able to access the same portions of the virtual address 
space, regardless of which processor executes the instructions. 
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Changes in the address space bounds are passed to server processors when the control 
thread is transferred to that machine. The server does not need this information until the 
control thread moves to the server . 7 The client processor includes the current address space 
bounds with the CAPC call packet. When the control thread moves to the server, the server 
adjusts its page maps and address space boundaries to match the boundaries presented in the 
client’s call message. 

When the virtual address space expands, the server adds new page table entries for the 
new pages. The server marks these pages resident on the client processor. If the server 
attempts to access these new pages, they will be demand paged from the client processor. 

Decreasing the address space requires additional work to ensure that the client and 
server views of the address space remain consistent. If the client shrinks and re-expands the 
data segment, it replaces those pages with zero fill-on-demand pages. This operation dis- 
cards any data on the affected pages. If the server has copies of those pages, it must also dis- 
card those pages. When shrinking the address space, the client must notify the server of the 
invalidated pages of the address space. 

The CLASP kernel saves a low water marker for the bounds of the address space. 

Address space reductions set this low water marker. The client kernel passes the current 

bounds and the low water bounds when it passes control to the server. The server kernel 

examines the low water marker to see if the client discarded any pages that the server knows 

about. If so, the server invalidates those pages. After resolving address space reductions, the 

7 This is a design decision based on our underlying process model. Our base system (SUN Unix) provides a single 
control thread in each address space. In a system with multiple control threads, a different scheme to modify the 
address space is required. A possible scheme might designate one or more of the processors as owner of pages not 
yet in the address space. Extensions of the address space would be processed through this processor in a manner 
similar to the current network page fault mechanism. 
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server uses the current address space information in the CAPC to expand its page table to 
the current size. The new pages are marked resident on the client. 


3.5.2. System Calls 

The UNIX system call uses state information stored in the kernel. Some system calls 
reference simple information, such as the current time. Others reference more complex state 
information, such as I/O descriptors. The CLASP architecture does not attempt to replicate 
this state information between the client and server kernels. The client and server share only 
information about the bounds of the user’s address space. 

Applications must make system calls on the processor holding the appropriate state 
information. Because the client processor holds all state information, system calls occur on 
the client processor. User programs access UNIX system calls through a collection of C sub- 
routines that package their arguments and trap to the UNIX kernel. An easy way to force 
system calls to a particular architecture is to compile these subroutines for that architecture. 
The current CLASP implementation uses this scheme to force system calls back to the client 
processor. Client calls to these subroutines execute on the client processor. Server calls to 
these subroutines generate CAPC calls from the server to the client processor, where the sys- 
tem call is executed. 

CLASP does not disallow system calls on the server processor. In many cases, it can be 
more efficient to perform system calls on the server. For example, assume an application 
that uses a file stored on the servers disks. In the current implementation, I/O operations on 
this file are processed on the client. If a routine on the server reads from the file, the data 
crosses the network twice: once using the network filesystem operations from the server to 
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the client and once from the client to the server as a CLASP network page fault. A modified 
version of the I/O related library routines can eliminate this overhead. These library rou- 
tines can direct the I/O operation to the appropriate kernel. Other systems have successfully 
used this technique to redirect system calls to the appropriate processor [27,37,77]. 

3.6. CAPCvsRPC 

The CLASP architecture improves on the RPC architecture in several respects. The 
best way to summarise these improvements is to note that CAPCs provide the same seman- 
tics as normal procedure calls, while RPCs provide a subset of the normal procedure inter- 
face. 

RPC client and server processes execute in separate address spaces. All data required 
for a remote procedure call must be passed as arguments to that routine and can not contain 
pointers. On the other hand, CLASP client and server processes execute in the same virtual 
address space. This allows routines to exchange data through global variables and to share 
pointer-based structures such as lists and trees. Thus, CLASP procedures can be used 
exactly like normal procedures. Converting a program to RPC usually involves rewriting it. 

There are situations where this shared address space can impact performance. When a 
CLASP server routine dereferences a pointer passed from the client, it may generate a net- 
work page fault to retrieve the appropriate page of the virtual address space. RPC systems 
dereference each pointer before packaging and transmitting the argument list. Therefore, the 
single RPC message contains the object described by the pointer and does not generate the 
extra network transaction possible in the CLASP system. Imagine the pathological case 
where all of a routine’s arguments are pointers, scattered through the address space. An 
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RPC system collects all the data and sends a single large message. A CLASP system might 
generate a network page fault for each argument. 

On the other hand, many common algorithms will be much faster in a CLASP-based 
system than using RPC. CLASP’s demand-paging approach moves only the arguments and 
data that are actually used between systems. As an example, binary searches through large 
sorted arrays can be efficient because the accessed portions of the array are transferred to the 
remote processor on demand instead of prepaging the entire array to the server. Pages, once 
transferred to the server, remain on the server until they are required by the client processor. 
Pages used only by the client remain on the client; pages used only by the server will be 
transferred to and remain on the server. Pages of data used by both processors migrate 
between hosts on demand. Demand-paging allows CLASP to support arbitrarily long argu- 
ment vectors. RPC systems, because they package all of the arguments into a single mes- 
sage, limit arguments to the maximum length of a network message. 

Another improvement that CLASP makes over RPC systems is the way remote pro- 
cedures are invoked. RPC systems use special calling sequences to access remote procedures. 
Some RPC implementations replicate this calling sequence at every place that invokes the 
remote routine [16]. Other implementations encapsulate these calling sequences into a local 
stub routine [3]. CLASP uses neither special calling sequences nor stub routines. Instead, the 
client uses the subroutine call instruction of the local architecture with the target address of 
the desired routine, regardless of whether it is local or remote. Similarly, the server uses the 
subroutine call instruction of its architecture with the target address of the desired routine. 
The CLASP kernel uses the virtual memory system to detect the transfer to routines that 
execute on the server. The special instructions used to transfer control in RPC systems are 
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replaced by functions in the CLASP kernel. 

Because RPC systems use special calling sequences to invoke routines on remote proces- 
sors, they do not provide transparent means to pass local and remote procedures as formal 
parameters to subroutines. CLASP allows applications to pass both local and remote func- 
tions as arguments to subroutines. The called subroutine does not require any special pro- 
cessing to handle both local and remote functions; it uses the same instruction sequence to 
invoke both function types. 

RPC systems have several advantages over CLASP. Because the RPC client and server 
execute in separate address spaces, one half can be changed without affecting the other. 
Thus, new RPC servers can be installed without changes to existing RPC clients . 8 Program 
changes in a CLASP-based system require that the program be relinked to incorporate the 
new code. 

RPC servers can be more secure than a CLASP program. Because the RPC server exe- 
cutes in its own address space, all interaction is through the RPC call interface; clients can 
not modify or destroy data stored in the RPC server. CLASP programs, because they exe- 
cute in a single address space, do not have this separation. 

RPC systems operate between systems with different internal data representations. 
Because all information passed between systems is contained in the paramters and results of 
subroutine calls, appropriate typing and conversion operations can be applied when the data 
is transferred between systems. CAPCs, because they transfer data between systems as pages 

* Some RPC implementations might require that no client of that RPC server be active when the new version is 
installed. Others allow active clients to continue with the old server while new clients are connected to the new 


server. 
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of uninterpreted data, only operate between systems that share internal data representations. 

Cross-Architecture Procedure Calls do a better job of emulating the normal procedure 
call than the Remote Procedure Call. Because CAPCs provide transparent access to routines 
on remote processors, they do not require changes to application code. RPCs usually require 
changes to application code. By eliminating the need to change application code, the CAPC 
eliminates extra programming costs. 
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CHAPTER 4. 

IMPLEMENTATION OVERVIEW 


This section of the thesis describes a prototype CLASP system. This implementation is 
based on Release 3.0 of the SUN UNIX Operating System [21]. The implementation uses 
SUN-3 workstations with Motorola 68020 processors. In the next sections, we describe the 
three components of our prototype: the CLASP loader, the CLASP daemon, and the operat- 
ing system kernel modifications. The CLASP loader assembles object files for several proces- 
sor architectures into a single executable file. The CLASP daemon cooperates with the 
modified kernel to instantiate server processes and record CLASP statistics. The kernel 
modifications detect and perform CAPC calls and returns; they also manage page residency 
for each CLASP process’s virtual address space. 


4.1. The CLASP Loader 

Our new loader is a modified version of the standard UNIX loader. The standard UNIX 
loader processes object files for a single architecture to generate an executable file. The new 
loader processes object files for several processor architectures to generate a multi- 
architecture executable file. This multi-architecture executable file differs from a single- 
architecture executable because it has several text segments. Standard UNIX executable files 
have a single text segment. Each of the CLASP text segments contains instructions for a 
different processor architecture. 
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The multi-architecture executable file is similar to a standard UNIX executable file. 
The text segment of the multi-architecture file is divided into several text subsegments; each 
of these subsegments contains instructions for a different processor architecture. The file 
header of a multi-architecture executable file contains extra information describing how the 
text segment is partitioned — which sections of the address space correspond to a particular 
processor architecture. 


4.1.1. Text Partitioning 

As the loader processes each object file, it determines the processor architecture from 
header information stored in that file. The loader places instructions for the each architec- 
ture in a contiguous segment of virtual memory. Segments begin on page boundaries; no sin- 
gle page contains information for more than one segment. This allows the CLASP kernel to 
detect non-local instruction references using the hardware supported page-level protections. 

The loader provides the kernel with a table that describes the multiple text subseg- 
ments. This table specifies the architecture for each subsegment, and its location and extant 
in the address space. Figure 4.1 illustrates the C structure clasphdr that defines this infor- 
mation. This structure is stored at the beginning of the executable file, just after the UNIX 
a. out header information. 


4.1.2. Replicated Code 

Short routines — like sqrtQ — are executed most effectively on the local processor. The 
network overhead to make a remote call and return makes it much more expensive to execute 
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#define MAXCLASPSEGS 4 /* max # architectures */ 

struct claspseg 

{ 

unsigned short cs_arch; 

unsigned long cs_relocation; 

unsigned long cs_first; 

unsigned long cs_last; 

>; 

struct clasphdr 

long c_nsegs; /* number of segments */ 

struct claspseg c_segment [MAXCLASPSEGS] ; 

>; 

Figure 4.1 

Clasp Header Structure 


/* architecture */ 

/* address shifts */ 

/* addr in 1st page */ 
/* addr in last page */ 


such routines on the remote processor. The network overhead overwhelms any speedup 
gained by executing on the remote processor. If the routine is called infrequently, this over- 
head is not a significant fraction of the total running time. However if the routine is called 
for each element of a large array, the network overhead is unacceptable. In some cases, both 
client and server make many calls to the same subroutine. 

These cases appear to require substantial overhead — it seems that one of the proces- 
sors must use remote operations to invoke the shared subroutine. A modified CLASP loader 
can eliminate this overhead by allowing multiple instances of the same function. 1 Each 
instance of the function executes on a different architecture. If both implementations are 
compiled from the same source, their execution will generate the same outputs. This follows 


1 At some point, the programmer still must decide which routines should be replicated. The loader does not 
make this decision. 
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from the properties discussed in Section 3.2. 

The loader symbol table allows multiple instances of symbols that reference addresses in 
the text segment. When the relocation information uses a text symbol, the loader attempts 
to use an instance of the label defined for the current architecture. If no instance of the rou- 
tine exists for the local architecture, the loader uses the instance for the remote architecture. 

The loader’s attempts to use local instances of replicated routines can fail when pro- 
cedures are passed as formal arguments to routines. The UNIX qsortfi) routine expects an 
array of elements, the size of the array, and a pointer to a comparison routine. Qsort invokes 
the comparison routine to determine if the array elements are in order. The call to qsort will 
be resolved with a local instance of the comparison routine (assuming it is replicated on both 
processors). If qsort executes on the remote processor, each call to the comparison routine 
will cause a CAPC call back to the client processor. At this time, we have no general solu- 
tion to this problem. 

4.2. Operating System Kernel Modifications 

This section describes how the CLASP kernel recognizes multi-architecture executable 
files, performs CAPC calls and returns, and transfers pages of the virtual address space 
between client and server systems. 

Although the described implementation is based on the SUN Unix 3.0 kernel, many 
details should be portable to other variations of the UNIX system, including the AT&T Sys- 
tem V standard [21,54]. 
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4.2.1. Recognizing CLASP Executable Files 

The UNIX kernel determines the type of an executable file from the a.out header. This 
header describes the architecture appropriate for the executable image. It also contains a 
magic number that describes how the image should be loaded into the virtual address space. 
Some magic numbers (e.g., OMAGIC) specify a process that runs with an impure text seg- 
ment; these processes can overwrite their instruction space. The ZMAGIC magic number 
specifies a process that is demand-paged from the executable file and shares its (read-only) 
text segment with other processes executing the same image. Our prototype defines a new 
magic number — CMAGIC — that specifies a demand-paged multi-architecture executable 
file. The CLASP loader uses the CMAGIC value in the a.out headers for executable images 
that it generates. 

The kernel loads executable images as part of the exec(2) system call. Our new kernel 
includes the CMAGIC value in the list of allowed magic numbers. 

As part of the ezec(2) call, the kernel builds a new address space from the executable 
image, replaces the existing address space, and starts the user process at a specified entry 
point. For CMAGIC files, the kernel instantiates a server process before beginning execution 
of the user process. 


4.2.2. Per-Process Structures 

The UNIX kernel maintains a u and proc structure for each process in the system. 2 


1 These structures define the user and process information for each active process in a UNIX system. Additional 
information on these structures can be found in the literature [62]. 
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These structures contain file descriptors, scheduling information, address space boundaries, 
and other state data for that process. CLASP processes require additional kernel informa- 
tion: the connection to the server processor, text subsegment boundaries, and other informa- 
tion. The u structure for each process now contains an extra field pointing to a claspData 
structure for that process. This structure is allocated from the kernel’s buffer pool dynami- 
cally when a CMAGIC process starts. When the process terminates, the space is returned to 
the buffer pool. The space requirements for non-CLASP processes are not increased. 3 Figure 
4.2 shows the C definition of the claspData structure. 

Our prototype also stores a copy of the clasphdr structure from the executable file in 
the u structure. It should be stored in the claspData structure. 


4.2.3. Virtual Memory 

Our prototype required two modifications to the virtual memory system. The first 
modification allows the CLASP kernel routines to protect instructions for non-local architec- 
tures so they can not be executed on the local processor. The second set of modifications pro- 
vides the paging functions to share the virtual address space between the client and server 
processes. 


3 The u structure is an integral number of pages. The claspData pointer uses space that is already allocated to 
that structure but is otherwise unused. 
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struct claspData 
{ 


struct 

wire_t cd_wire ; 

/* 

for network I/O */ 

struct 

claspNetwork cd_cn; 

/* 

the message */ 

struct 

claspNetwork cd_cn2; 

/* 

2-message ops */ 

lnt 

cd_havedata; 

/* 

cd_cn is full */ 

lnt 

cd_amsegment [MAXCLASPSEGS] ; 

/* 

segs i do */ 

long 

cd_textbase ; 

/* 

where they start */ 

long 

cd_database ; 



slze_t 

cd_dataadjust; 

/* 

used on server */ 

long 

cd_didshrink; 

/* 

if brkO shrunk */ 

slze_t 

cd_tshrink; 

/* 

should never shrink */ 

slze_t 

cd_dshrink; 

/* 

smallest dsize */ 

slze_t 

cd_s shrink; 

/* 

smallest ssize */ 

long 

cd flags; 

/* 

state flags */ 

#define 

CD READY Oxl 

/* 

in use */ 

#define 

CD_HAVEWIRE 0x2 

/* 

got one */ 

#define 

CD_C0NNECTED 0x4 

/* 

and connected */ 

/* 




* Instrumentation. 



*/ 




long 

cd_pageouts ; 

/* 

pages sent */ 

long 

cd_pageins ; 

/* 

pages yanked */ 

long 

cd_pagesize ; 

/* 

across wire */ 

long 

cd_localcalls ; 

/* 

i handle */ 

long 

cd_localrets ; 



long 

cd_netcalls ; 

/* 

to peer */ 

long 

cd_netrets ; 




>; 


Figure 4.2 

Dynamically Allocated structure for each CLASP process 


4. 2. 3.1. Page Protections 

The UNIX kernel already provides internal functions that protect individual pages of 
the address space. Existing kernel routines use this function to protect the text segment 
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against write access. A new function uses the single-page protection routine to protect a 
series of pages. The new function expects an address range and a protection and invokes the 
single-page function on each page in the specified range. When starting a CLASP process, 
the kernel uses this routine to protect the sections of the text segment that contain instruc- 
tions for the non-local architecture. 


4.2.3.2. Network Page Faults 

The second set of modifications to the UNIX kernel implement address space sharing 
between the client and server processors. During program execution, the pages of the virtual 
address space move between the client and server processor. When a page is resident on the 
server processor, the client processor can not access that page. Attempts to access the page 
generate a memory fault and the kernel must retrieve the page from the server processor. In 
this respect, the kernel must handle server-resident pages in the same fashion as pages stored 
on the swap device. The mechanism used to retrieve server-resident pages differs from that 
used to retrieve pages stored on the swap device. 

When a page is non-resident, the corresponding page table entry (PTE) contains infor- 
mation describing its location on the swap device. Some pages, which have never been 
resident, are not stored on the swap device. Instead, the first access to these pages causes the 
kernel to allocate a page filled with zeroes. These pages are called fill-on-demand pages. 
There are several kinds of fill-on-demand pages: fill with zero, fill from an arbitrary file, and 
fill from the executable image. The system pagein routine allocates and validates a physical 
page before invoking a fill-on-demand handler. The handler proceeds by loading the page 
with the appropriate contents. 
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Server-resident pages are marked as a new type of fill-on-demand page. These pages 
are marked CLASP fill-on-demand. The pagefault handler’s table of fill-on-demand 
handlers contains a new entry for the CLASP fill-on-demand page type. When processing a 
CLASP fill-on-demand fault, the pagein routine calls the new clasp_pagein() routine with the 
virtual address of the page and the length of that page. 

The clasp_pagein() routine issues a request to the server for the appropriate section of 
the address space. Clasp_pagein{) uses the communications descriptor stored in the clasp- 
Data structure to communicate with the server process. The server process replies with a 
message describing the page and the contents of the page . 4 The client . stores the retrieved 
page at the appropriate virtual address, marks the PTE as valid, and returns control to the 
system pagein handler. After sending a page to the client, the server invalidates its copy of 
the page: it frees the physical page frame and marks the appropriate PTE as CLASP fill- 
on-demand. 

When a page is retrieved from the server, the client marks the page modified, or dirty. 

This happens even though the client has not modified the page. Otherwise, the local pageout 

daemon might discard the copy of the page on the assumption that has not been modified 

since it was last written to the swap device. This is more straightforward than determining 

whether the remote processor modified the page. It does not require extra information in the 

PTE to store an extra modified since retrieved from remote bit. The cost for this decision is 

that a page may occasionally be written to the backing store twice. However, the extra disk 

transfer occurrs asycnchronously, it only affects the pages that aren’t modified again by the 

4 While the control thread is on the client, the server process sits in the clasp_rcv() routine. For network paging 
operations, elasp^revQ invokes the proper CLASP paging routines. When a control transfer message arrives, the 
claap^rcvQ routine returns to its caller — the clasp _Jault{) routine. 


76 


local processor, and should only occur for pages that fall out of the working set. 


4.2.4. Cross— Architecture Calls 

Applications make CAPC calls using the subroutine call mechanism of the local archi- 
tecture. For CAPCs, the target address is a section of the virtual address space protected 
against execution. When the processor attempts to fetch the first instruction of the called 
subroutine, the virtual memory hardware generates a protection violation signal. At this 
time, the kernel trapQ routine is called to handle the fault. 

The trap routine calls the new clasp_fault{) routine to handle protection violations. 
Clasp_fault{) determines whether the fault was caused by a CAPC call, a CAPC return, or is 
a stray memory reference. For stray memory references, clasp_fault() returns an indication to 
the trapQ routine to generate a segmentation violation signal for the user process. 
Clasp_faultQ decodes the instruction that generated the fault to determine whether it is a 
CAPC call or return. For CAPC calls and returns, the clasp_fault() routine packages the call 
information and transfers control to the remote processor. 

For CAPC calls, clasp Ja ult() determines the argument vector, the target address and 
the return address. The target address — the address of the called subroutine — is the 
address that caused the MMU to generate the fault. Machine-specific code examines the 
stack and also examines the intructions following the call instruction to determine the length 
and location of the argument vector. Clasp_faultQ stores this information in a claspPacket 
structure and sends it to the remote processor using the network descriptor stored in the 


claspData structure. 
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Clasp_Javlt() uses the clasp_xmit() routine to send the control transfer message to the 
remote processor. After sending the message, clasp-faultQ calls the clasp_rcv{) routine to 
receive the message that returns control to the local processor. The thread can come back as 
a nested CAPC call. It can also return to the client as a CAPC return. Clasp_rcv{) relinqu- 
ishes control of the CPU while awaiting messages. The process sleeps waiting for data to 
arrive on the network file descriptor . 5 

While awaiting the control transfer message, the server process is in the clasp_fault() 
routine . 6 When the message arrives, clasp_fault{) unpacks the message, builds a call frame on 
the (server) processor, and sets the appropriate values for the program counter, stack pointer, 
and other registers. For some cases, an argument register can be loaded with an appropriate 
value pointing to the arguments. In other cases, clasp^faultQ must make a copy of the argu- 
ment vector. After building the call frame, clasp_fault() returns through the kernel trap 
handler with an indication that the user process should be resumed with the new register 
values. At this time, the user program continues execution on the server processor. 


4.2.5. Cross-Architecture Returns 


On CAPC returns, clasp_fault{) takes the return values from the appropriate registers 
and stores them in the fields of the claspPacket passed from the client at call time. 
Clasp_faultQ then sends this packet back to the client processor. After sending the packet, 


* The kernel s/eep() routine causes the current process to await a specific event. The current process relinquishes 
control of the CPU and waits for the event. When the event occurs, another process will make a call to the ker- 
nel wakeupQ routine, which will move the sleeping process into the ready queue. 

' It is really in the elatp^revQ routine while waiting for the message. However, as soon as the message arrives, 
clasp_rcv() returns to clasp_fault{). Clasp_rev() also handles the server side of network paging requests. Client re- 
quests for pages in memory are directed to the clasp_pageoui() routine from within clasp_rev{). 
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clasp_Jault( ) waits for memory and control transfer requests. 

Upon receiving the claspPacket, the client unpacks the return values and loads the 
appropriate registers for the client architecture. Clasp_faultQ sets up the appropriate register 
values for returns in the same fashion it sets up registers for calls. 

4.3. The CLASP Daemon 

The CLASP daemon, claspd, listens on a TCP/IP socket for connections from client 
processes . 7 When it receives a connection, claspd forks a child process to act as server for that 
client. After spawning the server process, claspd awaits further service requests. The client 
and its server process communicate independently from the CLASP daemon. The child pro- 
cess uses a new CLASP-specific system call to become a server. This system call accepts the 
file descriptor of the network connection to the client as a parameter. 

The claspdQ system call allocates a claspData structure for the current process. The 
descriptor for the network connection is stored in this structure. Claspd then collects infor- 
mation across the network from the client process. This information includes the text seg- 
ment partitioning and a copy of the text segment. The CLASP kernel code returns to user 
mode after arranging for an immediate protection trap and setting a flag for clasp_fault(). 

The system trap handler calls clasp_fault{ ) when the user code generates the protection 
violation. ClaspJaultQ examines the flag set by the claspdQ system call to see that this is a 
server initialization fault. Instead of sending a message to the client process, clasp_faultQ 
waits for a control transfer message from the client. 

7 TCP/IP is a stream-oriented protocol that provides in-order, guaranteed delivery communications between two 
endpoints [73]. 
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4.4. CLASP Algorithms and Protocols 

The next several sections describe the protocols for choosing a compute server, establish- 
ing contact with a compute server, transferring control between servers (the client is also a 
server), and transferring memory between servers. The different structures for control and 
memory transfer are part of a single network structure. The examples in this chapter include 
only the relevant sections of the network structure. 

4.4.1. Server Availability and Selection 

A new kernel table is used to select a server processor. New system calls allow claspd to 
clear, replace entries, and append entries to this table. At startup time, claspd reads the file 
/« st /local/ etc/ claspd.config for profiling, logging, and server address information. Appendix 
D describes the syntax of the claspd.config file and includes an example configuration file. 

After determining an address for the server, the CLASP kernel routines try to establish 
a connection to that address. If the connection fails, the client process is aborted. 


4.4.2. Initiating a Dialogue with a Compute Server 

CLASP clients establish connections to the required server processors when the client 
process is instantiated. In our implementation, the operating system establishes these con- 
nections as part of the exec{ 2) system call. Once a server process has been started, the client 
process initializes the server by sending a description of the address space. This description 
includes the address space bounds and the text subsegment locations, sizes, and architectures. 
After this information is transmitted, the client process sends a copy of the text segment 
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across the network to the server. 8 

Our prototype does not try to connect to alternate servers if the first server does not 
respond. Our implementation does not provide a protocol for a server to deny service on a 
selective basis. 


4.4.3. Calling Procedures on a Compute Server 

Clasp_fault() determines the target address for remote calls, the location and length of 
the argument vector, the return address, and the stack bounds. This information is stored in 
a claspPacket structure. Figure 4.4 shows the C declaration for this structure. 

Clasp_fauli{) stores a copy of the claspPacket structure on the user stack. It builds a 
message to the server that contains the address of the claspPacket on the stack and a copy 


struct initlate_data 

{ 

struct clasphdr cnx_claspHdr ; 
long cnx_caller_segments [MAXCLASPSEGS + 1] ; 
long cnx_callee_segments [MAXCLASPSEGS + 1] ; 

long cnx_textbase; 

long cnx_database; 

>; 


/* from a. out */ 

/* client Is */ 

/* server Is */ 

/* segment bases */ 


Figure 4.3 

Data Transferred to Start A Server 
[part of the claspNetwork structure] 


* Pushing a copy of the entire text segment across the network can be expensive. Some processes have as much as 
a megabyte of instruction space. A future implementation might pass file handles (like the NFS mode) so the 
server can demand page portions of the address space from the file on the client processor [15] . 


81 


struct claspPacket 

{ 

long cp_actlon ; 

/* 

* values used In a call 
*/ 

caddr_t cp_subroutlne ; 
caddr_t cp_sp; 
caddr_t cp_argllst; 
long cp_arglen; 

/* 

* values used In a return 
*/ 

caddr_t cp_return; 
caddr_t cp_usp; 
unsigned long cp_rO; 

unsigned long cp_rl ; 

/* 

* server end of a call uses this 
*/ 

struct claspPacket *cp_lastp; 

>; 


/* call, return, etc */ 


/* address to call */ 

/* where server can start */ 
/* base of arg vector */ 

/* length of arg vector */ 


/* return address */ 

/* user SP after return */ 

/* return 0 */ 

/* return 1 */ 

for temp storage 

/* for nested cross-calls */ 


Figure 4.4 

CAPC Information Packet 


of the claspPacket. The controlxfer structure, depicted in figure 4.5, is part of the 
larger claspNetwork structure. Other fields in the claspNetwork structure contain the 
current address space boundaries; these fields are loaded before the message is sent to the 
remote processor. 


4.4.4. Returning from Procedures on a Compute Server 

For returns, clasp_fault() uses the existing claspPacket from the CAPC call. The 
cp_action field is changed from call to return, and the return value fields are loaded with 
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/* 

* The claspNetwork combined structure contains fields describing 

* the extent of the address space. 

*/ 

struct controlxfer < 

long cnx_ctlxf ertype ; /* call or return */ 

struct claspPacket *cnx_cpp; /* pointer to CAPC frame */ 

struct claspPacket cnx_f astcapc ; /* copy of CAPC frame */ 

>; 


Figure 4.5 

CAPC Control Transfer Information 


values from the appropriate processor registers. 

Clasp_fault[) then follows the same steps to send this claspPacket back to the client as 
it would to send a call message to the client: address space bounds are loaded into the 
claspNetwork structure and the message is written on the network descriptor in the 
claspData structure. 


4.4.5. Memory Transfers Between Clients and Servers 

The CLASP prototype memory system uses a simple model to maintain coherency in 
the virtual address space: each page of the address space resides on exactly one host. If one 
CPU needs a page that resides on another host, the page is demand-paged from the remote 
processor to the local processor. The local processor — the one that wants the page — is the 
client. The non-local processor that currently holds the page is the server. A processor acts 
as both client and server at various times through the lifetime of an application process. 
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When the CPU attempts to access a page that is not resident in main memory, the 
MMU generates a page fault. Non-resident pages can be retrieved from several different loca- 
tions. Some pages are retrieved from the local backing store (e.g., the swap space of the local 
processor). Other pages reside on remote processors and must be retrieved across the net- 
work. In both cases, the page table entry describes where the page is stored and how to 
retrieve the page. 

The client CLASP kernel marks pages resident on the server as fill-on-demand-clasp. 
This allowed us to implement the shared virtual address space without changing the width or 
bit assignments of fields in the page tables. The kernel pagein routines treat fill-on- 
demand-clasp similarly to fill-on-demand-zero. Instead of zeroing a page and validating it 
for the user, page faults on fill-on-demand-clasp pages cause a call to a CLASP-specific 
pagein routine which retrieves the page from the remote processor and places it in the page 
frame allocated by the normal pagein routine. The changes to the normal pagein routine are 
limited to an additional case in the switch statement that handles fill-on-demand pages. 

The client sends MEMGET requests to the server to retrieve pages that reside on the 
server. These MEMGET packets contain the above memxf er structures to describe the pages 
requested. Servers reply with a MEMPUT packet that describes the pages being returned 
and follow that packet with the data of the page. If the client and server have different page 
sizes, the MEMPUT packet might describe a different (e.g., larger) block of memory. A client 
with 512 byte pages making a request to a server with 1024 byte pages would receive two 512 
byte pages. 9 

' These differences should be resolved when a CLASP process begins execution. The client and server should 
agree on a network page size that meets their individual requirements for local pages. All network paging opera- 
tions should be done in units of this agreed page size. 
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struct memxfer 

{ 

caddr_t cnx_base; /* base In vaddr */ 

caddr_t cnx_length; /* byte count */ 

>; 


Figure 4.6 

Information Sent for Memory Transfers 


Once pages are retrieved from the server, the client marks them dirty, or modified, to 
inform the pageout daemon that these pages have been changed since the last time they were 
written to the local swapping device. This assumes that the server modified the page. While 
it might not always be true, the alternative was to add several additional dirty bits to the 
page table entry — one for each backing store that might hold the page. 

This simple memory model made our prototype easy to implement. However, the 
model limits performance by keeping only one copy of any page in the address space. If a 
processor has a local copy of a given page, the overhead of a page fault across the network 
can be avoided. One problem with replicating pages on each processor is maintaining the 
coherency of the virtual address space. The replication of read-only data is a simple and 
obvious way to improve performance. 10 We describe some other work that supports multiple 
instances of pages in the address space in section 6.2, under Further Performance Optimiza- 
tions. 


10 Our prototype keeps copies of the text (code) segment on both machines. The CLASP text-segment is filled 
with read-only data (e.g., the instructions). 
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CHAPTER 5. 

PERFORMANCE OF THE IMPLEMENTATION 


CAPC allows an application program to be partitioned so that some routines execute on 
a faster server processor to decrease the running time of the program. Partitioning an appli- 
cations program always introduces some overhead: CAPC subroutine calls and returns are 
slower than local subroutine calls and returns. Data residency also affects the partitioning’s 
overhead. Some pages are accessed by the routines on the client; others are accessed only by 
server routines. Some pages are accessed by both client and server routines. Pages accessed 
by both processors must be moved to the appropriate processor when needed. 

To overcome this overhead, the server must be faster than the client processor. The 
breakeven point can be derived from the paging behavior, CAPC calling patterns, and speed 
differential of the two processors. This breakeven point is different for each program and can 
vary within a single program depending on how it is partitioned. 

In this chapter, we discuss types of algorithms that will perform well under the CLASP 
system. We also discuss an existing model that characterizes paging behavior [32,33]. We 
present the results of benchmarks to determine the costs of our system. These costs include 
remote call overhead and network paging overhead. In appendix B, we show the results of 
several benchmark programs under the CLASP system. Section 5.3 compares our empirical 
results with those predicted by section 5.1. The chapter closes with a discussion of several 
mechanisms for partitioning applications between client and server systems. 



5.1. Theoretical Performance Expectations 


To determine whether part of a program should be moved to a server processor, an 
appropriate question to ask is: is the execution speedup greater than the communications 
costs? If data transmission costs are greater than the possible speedup, the problem should 
not be moved to the server processor. 

Several factors affect the performance of a partitioned application. Programs where the 
execution costs grow faster than the communications costs to move data between client and 
server quickly overcome the communications overhead. Algorithms that access portions of a 
data structure, such as tree searches, are another class of algorithms that can yield improved 
performance when partitioned between processors. 

5.1.1. Algorithms Appropriate for the CLASP architecture 

Algorithms for solving linear systems are a good example of problems where improved 
execution time on the server recovers the communications time between the client and server. 
Linear systems of order n comprise an nxn matrix. The cost to move this problem to a 
remote processor across a network is 0(n*n). The time to factor this matrix is 0(n*n*n). 
The breakeven point occurs when the speedup on the remote processor matches the communi- 
cations cost to move the data to the remote processor. If A(n) is the savings in execution and 
T(n) is the communications cost, our breakeven point occurs when we satisfy the following 
equation. 


T(n)=A(n) 
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For an applications using gaussian elimination to factor matrices of order n, the 
appropriate equation is : 1 

K x n*n = K 2 n*n*n 

The exact value of n that satisfies this equation depends on the constants. These con- 
stants are determined by the network speed, client processor speed, and server processor 
speed. 

To determine if there exists an n where the problem should be moved to the processor, 
we examine the inequality: 

»—oo Afn) 

If this limit is less than one, there will be some problem size n that executes faster in a 
partitioned environment. 

We do not want to give the impression that only higher order algorithms are applicable 
to our architecture. A number of data structures have search times smaller than Ofn ). 
Trees searches execute in time Oflog n). These searches do not require the entire data struc- 
ture to be resident on the local processor. The Oflog n) probes of a tree search will move at 
most Oflog n) pages from the remote processor. Additional searches, which often probe the 
same initial nodes of the tree, will generate fewer page faults. Insertions, balancing, and 
other tree operations can often execute with only portions of the data structure resident. 

Other data structures that require less than Ofn) time to manipulate are hash tables, 
queues, lists, and heaps. For these data structures, the communications costs are a function 



1 For this example, we have dropped the lower order terms from the algorithm costs. 
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of the probes into the structure. 

5.1.2. Localized Data 

Some data structures are accessed only by several routines. If all of these routines are 
implemented on the same processor, there are never any communications costs to access that 

data. 2 

In such cases, the costs to move this portion of an application to the program are 
related only to the frequency and duration of the calls to those subroutines. If the subrou- 
tines execute for more than our CAPC call/return overhead, we expect improved perfor- 
mance by moving them to a faster processor. The exact client/server speed ratio needed to 
compensate for the CAPC call/return overhead depends on the time for the subroutine call 
on the client. 

5.1.3. Paging Patterns 

In 1968, Peter Denning introduced the working set model to help manage page traffic in 
virtual memory systems. The working set model uses recent page access history to predict 
the short term memory needs of a program. The objective is to keep a working set of pages 
resident in memory and increase the average instruction burst between page faults. The 
working set model is based on the observation that programs show localized access patterns. 
Working sets exhibit slow drift behavior; the working set changes gradually over time [31]. 


* This ignores the boundary condition when we start the process and the entire address space is resident on the 
client or the client’s swapping device. 
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By 1974, Denning and others determined that the slow drift concept was wrong [32]. 
While programs did have phases which showed slow drift behavior, these programs also 
displayed disruptive transitions between phases. Most phases used almost completely 
different sets of pages, or locality aete[24,25,33,51,53,63]. Kahn found the following about 
phases and transitions between phases: [53] 

• Phases covered 98 percent of the virtual time. 

• 40 percent to 50 percent of the working set page faults occurred in transition 
periods. Thus, about half of the paging occurred in 2 percent of the virtual 
time. 

• The same phases were observed by the working set policy over wide ranges of 
its control parameters. 

• Fault rates in transitions were 100 to 1000 times higher than fault rates in 
phases. 

Other observations indicate that approximately 90 percent of the virtual time is spent in long 
(at least 100,000 memory references) phases. These long phases account for only 10% of the 
recognized phases, the other phases are fleeting and embedded within transition periods. 

To model phases and the transitions between phases, Denning built a macro-model 
using semi-Markov chains. The states in this model correspond to phases and their locality 
sets. The holding time of each state corresponds to the phase length for that locality set. 
Within each state, Denning used existing micro-models to generate access patterns across the 
pages in the locality set. Denning found that this macro-model followed the behavior of real 
programs better than the existing models. 
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5.1.4. Performance Expectations 

We expect that the CLASP architecture will work well for a number of applications. 
These applications will have one or more of the following properties: 

• algorithms where the execution costs grow faster than the communications 
costs. An example is the solution of Ax = b, which requires 0(n*n) communi- 
cations and 0(n*n*n) execution time. 

• algorithms which exhibit high degrees of locality. 

• access methods, such as hashing and tree searching, which move small parts of 
a larger data structure. 

• subroutines that encapsulate access to data structures. These data structures 
will not move between processors, so the communications costs are only the 
procedure call overhead. 

Where communications and execution costs are of different magnitudes, the advantages 
are apparent. Where both costs are of the same magnitude, the coefficients become more 
important. In both cases, the exact breakeven point depends on the particular application 
and its calling patterns. In some cases, demand-paging saves two network faults — such as 
when the results of one remote operation are passed directly to another remote operation. 

5.2. Empirical Results 

We ran a series of benchmark programs with our CLASP kernel to obtain a measure of 
its performance. Some benchmarks provided us with the overhead of the cape mechanism 
and paging costs between client and server. Several benchmark programs, acquired from 
other sources, were used to generate information on the frequency of paging traffic between 
client and server systems. In this section, we discuss the benchmarks used to determine the 
CAPC call/return overhead and the network paging costs. Appendix B contains performance 
data for other benchmarks and looks at timing, speedup potential, and paging behavior of 
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those benchmarks. The next section (5.5) compares the observed paging patterns with the 
behavior we predicted in section 5.1. 

Remote operations — both calls and data accesses — are almost always significantly 
more expensive than local calls. The network overhead accounts for most of this difference. 
We can divide the network overhead into two pieces: latency and bandwidth. For smaller 
messages (such as control transfer packets), network latency dominates the overhead. Other 
research provides insights towards developing low-latency, high-bandwidth networks 
between processors [50,78]. 


5.2.1. Remote Call and Paging Costs 

Figure 5.1 contains the times for CAPC calls and returns in our prototype. The CAPC 
overhead number represents the time for the client to invoke a subroutine on the server with 
no arguments and for that subroutine to return. The number is an average across 10,000 
invocations of the subroutine. Timings of individual calls — to obtain minimum, maximum, 
and standard deviation figures — was not possible with our hardware. The clock on our sys- 
tems ticks at 50 Hz, the smallest interval we can measure is 20 milliseconds. 

Figure 5.1 also contains the time for network paging. We used two different programs 
to generate this data. The first page fault program is based on the CAPC call overhead pro- 
gram. In this version, the calling routine accesses a global variable once during each iteration 
of the loop. The server routine accesses this same variable once during each call. This causes 
2 page faults for each CAPC call/return pair. We determined paging costs by subtracting 
the known CAPC call overhead. From this program, we can determine the time for a pair of 
page faults. 
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The second program alternates calls to subroutines on the client and server that step 
through a large array. We ran this test with arrays ranging from 100 through 400 pages — 
8192 Kbytes through 3276 Kbytes. Arrays larger than approximately 2200 Kbytes filled the 
available physical memory and introduced other factors into the times; the desired page 
would not be resident in main memory and had to be retrieved from the swapping device. 
The numbers in figure 5.1 reflect the times for 2000 Kbyte arrays. Because this program gen- 
erates long strings of page faults in one direction, we can time those strings to determine the 
times to send or receive pages. This allows us to break down the round-trip costs obtained in 
the first paging benchmark. 

These benchmarks were run using the loopback interface to the same processor. These 
timings show that the process spends large amounts of time in the system kernel. We believe 
that most of this time is TCP/IP protocol overhead. With 53 milliseconds per page fault, 
our prototype kernel can process 18.8 pages or 154,000 bytes per second. Additional bench- 
marks showed that a TCP socket could only move 297,000 bytes per second on our system. 
Our prototype provides more than half of the throughput with the current network protocols. 
Synchronous page requests and page transfers account for the lost page bandwidth. 

Figure 5.2 shows the times for empty procedure calls using CAPC, Sun RPC (both UDP 
and TCP transport mechanisms), and Xerox Courier. In all but one case, the times are aver- 
ages across 10,000 calls. The times for Courier with the standard kernel for 1,000 calls. The 
Courier times for the standard kernel are more than an order of magnitude slower than the 
other mechanisms. This is caused by the SUN TCP implementation. Instead of flooding the 
network with small TCP packets, the SUN TCP code delays small packets so it can combine 
them into a larger packet. If no further data arrives, a timer signals the TCP code to send 
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CAPC Test Case 

Client 
svstem time 

Server 
svstem time 

Wall 

time 

CAPC Call Overhead 

4.46 

4.45 

8.98 

CAPC -l- 2 page transfers 

56.74 

56.70 

115.44 

2 page transfers 

52.06 


106.42 

1 Page Transfer 

26.03 


53.21 

CAPC + 250 page xfers 




avg/page 

26.34 

26.86 

53.42 

client- > server avg/page 

30.36 


53.52 

server- > client avg/ page 

22.28 


52.92 


Figure 5.1 
CAPC Overheads 
[all times in milliseconds] 

the small packet. From these times, we can see that the timer fires every 200 milliseconds. 


For these tests, the Courier Code seuus messages that fall bclon this threshold. 
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the kernel to lower this threshold and re-executed the Courier benchmarks. 3 This small 


packet threshold does not affect the timings for the other benchmarks because they send 
larger packets between client and server. 


F or small argument vectors, the CAPC call packet contains a copy of the argument vec- 
tor. Larger argument vectors are passed by pointer; the server will demand page the argu- 
ment vector across the network. Our prototype sends up to 64 bytes of arguments in the 
CAPC call packet. For these small argument vectors, a CAPC costs approximately 9 mil- 
liseconds. For larger argument lists, the call time is 9 milliseconds plus the time to move the 
appropriate stack pages to the server. Because these stack pages are required on the client 
after the called subroutine returns, they must be paged back from the server routine. In 


* Other TCP implementations we have seen do not have this small packet threshold. For example, the 4.2 BSD 
kernel does not hold these small messages. Also, it is worth noting that most Courier calls will be larger than the 
10 byte small-message threshold. Return messages, if they are simple integer values, fall below the threshold. 
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Mechanism 

User Time 

System Time 

Wall Time 

CAPC 

0.0 

4.46 

8.98 

Sun RPC (udp) 

0.432 

3.076 

7.942 

Sun RPC (tcp) 

0.846 

3.118 

8.014 

Courier (stock kernel) 

0.0 

0.020 

399.960 

Courier (modified ker- 
nel) 

0.48 

5.26 

11.78 


Figure 5.2 
Empty Call Costs 

' [all times in milliseconds] 

most cases, argument vectors larger than 64 bytes will take approximately 115 milliseconds. 
If the argument list crosses page boundaries (the sun-3 hardware uses 8 kbyte pages), the 
costs go up by another 2 page faults — another 100 milliseconds. Figure 5.3 shows the aver- 
age costs for CAPC calls with argument vectors ranging from 0 to 1024 bytes. These times 
are to set up, execute, and return from the remote subroutine. 

Since most procedure calls have small argument lists it is sensible to spend our efforts 
making the most frequent case execute quickly. Code analysis done for RISC machines has 
shown that, in a UNIX environment, many procedure calls have fewer than 6 arguments. 
These procedures often account for more than one-half of the dynamically executed pro- 
cedure calls [35,70]. Our prototype, which provides 64 bytes of fast arguments, handles 16 4- 
byte arguments before falling back to the slower call mechanism. Therefore, our optimized 
CAPC calls for small argument vectors should handle most procedure calls. Larger argument 
lists are processed more slowly, but they make up a small percentage of the subroutine calls. 
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Call+Return 

(milliseconds) 



Argument Vector Size 
(bytes) 


Figure 5.3 

CAPC Overheads for Different Argument Lengths 


5.3. Comparing the Facts to the Theory 

The LINPACK benchmark described in Appendix B showed very good performance in 
our system. This benchmark moves data of size 0(n*n) from client to server and performs 
0(n*n*n) operations on that data. The breakeven point for this benchmark came for sys- 
tems of size 59. For a problem of this size, the server had to execute at 17 times the speed of 
the client processor to compensate for the network overhead. For matrices of order 81, the 
server had to be only twice as fast as the client to compensate for the network overhead. 

This benchmark used matrices dimensioned at for 200x200 systems. This over- 
allocation generated extra page traffic by removing some of the locality within the array. 
Another version of the benchmark, using matrices dimensioned to the exact size of the sys- 
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tern, shows a better breakeven point. The speed differential can compensate for the overhead 
on a 42x42 system. A server executing twice as fast as the client breaks even on a 54x54 sys- 
tem. 

A second benchmark, the compress utility did not show an improvement when parti- 
tioned between client and server [87]. This program operates as a filter on its input data. 
For the test we ran, the network overhead was larger than the total execution time for a sin- 
gle architecture version of the program. 

5.4. Considerations For Partitioning Applications 

Two factors affect the placement of a routine: the cost to execute a CAPC call and the 
cost to demand page the required memory to the remote processor. Small subroutines often 
do not execute enough instructions for the client/server speed differential to overcome the 
CAPC overhead. Other subroutines may execute enough instructions to make up the CAPC 
overhead, but their data access patterns may cause an excessive number of page faults. 

In terms of Dennings model, we want to partition our program so that cross- 
architecture calls have a close correlation with transitions between phases. Short procedure 
calls may generate extra paging traffic and disrupt the phase/transition page fault patterns. 

In this section of the thesis, we discuss how these factors can affect performance. The 
next section presents an existing algorithm for partitioning applications between client and 


server processors. 
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5.4.1. Frequency and Duration of Calls 

In our implementation, a simple CAPC call and return costs approximately 9 mil- 
liseconds. Local subroutine calls, using only several microseconds, can be considered free 
when compared to 9 milliseconds. If the routine executes locally in less than 9 milliseconds, a 
remote call will always be slower, regardless of the speed of the remote CPU. If the local 
time is greater than 9 milliseconds, the breakeven point depends on the relative speeds of the 
processors (and the data residency). 

Ignoring data residency, a subroutine that executes in 18 millisecond on the workstation 
can be moved to a server that executes twice as fast as the workstation. The server will exe- 
cute the subroutine in 9 milliseconds, plus the 9 millisecond CAPC overhead, and achieve the 
same total time as the workstation invocation. 

In practice, data residency affects the breakeven point by introducing paging overhead 
to move the data to the remote processor. Appendix B shows several program examples, how 
they perform in uniprocessor mode and under CAPC, and what speed differentials are 
required to break even for different problem sizes. 

The frequency of calls to a subroutine affects its placement. If a subroutine is called 
only a few times during the execution of a program, the overhead of a CAPC call has little 
impact on the total running time of the program. An extra several hundred milliseconds has 
little affect when a program executes for minutes or hours. However when calling routines 
hundreds or thousands of times during the execution of a program, any additional overhead 
becomes significant. Section 5.5 describes techniques to partition routines between client and 
server systems to minimize this overhead. 
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An alternate approach is to replicate these frequently called subroutines on both archi- 
tectures. For long-executing routines (e.g., factoring large matrices), this is not practical. 
Replication of long-executing routines is counter-productive; we want these routines to exe- 
cute on the faster processor. For short subroutines — like sqrt() or strcmp() — replication is 
important. Without replication, the overhead to invoke remote instances of these routines 
can overwhelm any performance improvement gained by moving long-executing subroutines 
to the faster processor. 


5.4.2. Data Residency- 

In our prototype, each page exists on exactly one of the processors. If a pair of pro- 
cedures on different processors alternately access a page, that page bounces between the pro- 
cessors. We can reduce or eliminate this effect by placing both procedures on the same pro- 
cessor. 

Page replication schemes eliminate the problem when neither processor modifies the 
shared data. Each processor maintains a copy of the page and allows read access by subrou- 
tines executing on that processor. If a subroutine attempts to write on a replicated page, the 
other copy must be updated or invalidated. Again, the time for these operations raises the 
overhead associated with splitting these routines. But the overhead is often significantly 
lower than a non-replicated environment. 
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5.5. Determining Partitionings 

Subroutine call frequency and data residency are factors in determining how to partition 
an application. These factors, by themselves, provide information about costs of a partition- 
ing. However, they do not determine partitionings. 

The next two sections discuss several approaches to partitioning applications. The first 
section discusses several tools that can be used to provide information about call patterns and 
the duration of procedure calls. The data gathered from these tools can be used to make 
decisions. It can also be used as input to more formal partitioning schemes. One of these 
schemes is discussed in section 5.5.2. 


5.5.1. Heuristic Partitioning Tools 

Profiling tools are useful for identifying where a program spends most of its time. The 
UNIX utilities prof and gprof provide different types of profiling. Prof provides a summary of 
the total time spent in a routine and the number of times it was invoked. Gprof provides this 
information and adds the calling patterns between routines. For each routine, gprof reports 
the calling and called subroutines. 

An easy way to determine a partitioning is to execute the program with the profiling 
tools. From this data, long running routines or sets of routines (e.g. a locality set of subrou- 
tines) can be identified and moved to the server. 
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5.5.2. Theoretical Partitioning Methods 

Other researchers have generated algorithms and heuristics for partitioning applications 
in a distributed environment. These approaches use mathematical tools to determine the 
partitioning [38,39,79]. 

In their 1978 paper, Stone and Bokhari use graph theory to determine subroutine place- 
ment in distributed systems [79]. Each subroutine in the application is a vertex in their 
graph. The edges between vertices represent the calling patterns between routines. Each 
edge is given a weight corresponding to the communications costs if the two vertices (subrou- 
tines) are on different processors. Two additional vertices, representing the processors, are 
added to the graph. From each processor vertex, edges are drawn to all subroutine vertices. 
These edges are assigned weights that correspond to the execution time for that subroutine 
on the other processor. Because the edges represent the costs if two routines are on separate 
processors, the weight is the execution time for that routine on the remote processor. After 
generating this graph, they generate a minimal cutset of the that graph. The processor ver- 
tices will be in different subgraphs. The two subgraphs contain the subroutines to be loaded 
on each processor. Stone shows that the minimal cutset generates an optimal placement for 
the subroutines [79]. 

Edge weights consider how often one procedure calls another and the data transfered for 
each call. In our shared memory model, we must also consider routines that interact through 
shared global variables. This consideration adds edges to the graph for subroutines that do 
not call each other, but do access the same variables. 

We want to bind certain routines to specific processors. For example, most system calls 
must execute on the client processor — where the appropriate kernel state information 
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resides. We can bind a procedure to a specific processor by adding an infinite weight edge 
between the vertices for the desired processor and the appropriate subroutine. In the the 
processor /node context, this indicates that the execution time of the subroutine on the 
remote processor is infinite, or that it can not execute on the remote processor. In the cutset 
context, this edge will never be in a minimal cutset. A cutset with this edge would have 
infinite weight. 

These mathematical models can yield optimal or near optimal partitionings for applica- 
tions. Some of their input data can be gathered from static analysis of programs. The tools 
discussed in section 5.5.1 can provide additional information for calculating edge weights. 
Further research into combinations of these tools could provide automatic partitioning 


schemes that combine all of these tools. 
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CHAPTER 8. 
SUMMARY 


This thesis has introduced CLASP, a new software architecture for sharing processor 
resources. CLASP maps the traditional UNIX process model onto a new foundation. CLASP 
provides a transparent mechanism for transferring control between processors and allows sec- 
tions of an application program to execute on the most appropriate architecture. This con- 
trol transfer is implemented by the Cross-Architecture Procedure Call or CAPC. CAPCs 
allow existing applications to be partitioned between processors without making source code 
changes. 

Section 6.1 discusses some additional research suggested by our investigations. Some 
approaches to reduce the overhead of the CAPC are discussed in section 6,2. The chapter 
closes with a summary of our results. 

6.1. Further Research 

There are a number of additional research topics related to CLASP and the CAPC. 
Some of this research is concerned with improving CAPC performance: using faster network 
protocols and reducing network paging traffic. These research areas are discussed in section 

6.2, Future Performance Optimizations. 

Other research to add new features to CLASP and to apply existing tools to CLASP sys- 
tems includes: multiple compute servers, multi-thread computations, multi-architecture 
debugging, asynchronous traps, I/O operations on servers, process migration, operating sys- 
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tem independence, and automatic program partitioning. Each of these is described in more 
detail in the following sections. 

6.1.1. Multiple Servers 

Our prototype supports two architecture CLASP programs. The extension to three or 
more architectures requires additional work in communications between the different proces- 
sors. 

As the number of architectures (and processors) climbs, the replicated address space 
work of Kai Li becomes a more important factor to reduce the cost of network paging [60,61], 
In these situations, a process must determine which CPU has the copy of the page — in addi- 
tion to moving it to the local host. 


6.1.2. Multi-Thread Computations 

A number of existing systems provide multiple control threads within the same address 
space [22,28,30,74]. We would like to see a combination of these systems and our CAPC sys- 
tem. Such a combination would allow programs to use the most appropriate mechanism for 
performance improvement — parallelism or fast sequential processing — within a single 
application. 

Our current prototype uses the single-thread nature of the Sun UNIX process to stream- 
line some operations. The same agent on the server processes page requests and control 
transfer requests. A multi-thread implementation would need to partition the address space 
and control flow operations. Our prototype deferes the propogation of changes in the address 
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space until it passes the control thread to the remote processor. A multiple-thread imple- 
mentation requires a different address space propogation mechanism. 


6.1.3. Debugging 

The CLASP system introduces several problems for debugging systems. Symbolic 
debuggers must now understand the instruction and calling sequences for different processor 
architectures. The system debugging facilities (e.g., the UNIX ptrace(2) system call) must be 
able to manipulate the control thread of a program when it is on a remote processor. 


6.1.4. Asynchronous Traps 

The UNIX signal mechanism provides a means to transfer control to a specific routine 
on the occurrence of specific events. Our prototype does not address the problem of how 
these signals should be processed when the control thread is on the remote processor. 


6.1.5. I/O Operations on the Server 

Our prototype performs all system calls, including I/O operations, on the client system. 
We make this restriction because the existing file descriptors are stored in the kernel on the 
client system. As long as the correct file descriptors are presented to each system, it should 
be possible to perform I/O operations on both systems. One approach that supports this 
operation is to modify the system call templates to select the proper system for the system 
call. This technique has been used in systems like the Newcastle Connection to redirect sys- 
tem calls to the kernel that holds the appropriate state information [27]. 
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6.1.6. Graceful Process Migration 

Because the server kernel only requires information about the address space of a process, 
it should be able to move all of this state to another processor. There are no file descriptors 
to move between server processors. Further work with the CLASP system should produce 
mechanisms that allow servers to re-direct clients to other processors of the same architec- 
ture. This can be used to limit the load on a particular server. It might also be used when 
rebooting a server; existing clients could be moved to other processors. 


6.1.7. Operating System Independence 

Because the CLASP server maintains only address space information, we should be able 
to implement servers with an open systems architecture. Clients running the UNIX operating 
system might communicate with servers on other operating systems such as DEC’s VMS, 
CDC’s NOS, the Stanford V kernel, CMU’s MACH, and other operating systems. The rou- 
tines that execute on the server do not access system functions, they only use the processor to 
execute instruction sequences. 


6.1.8. Automatic Partitioning 

The algorithms described in section 5.5 partition programs to reduce the communica- 
tions costs and improve the performance of a program. Compilers already generate data 
dependency information and can generate control flow information. Software generation sys- 
tems (compilers and tools like the UNIX make utility) could use this information to partition 
applications without user interaction [40]. 
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6.2. Future Performance Optimizations 

CLASP systems rely on an underlying network communications system to transfer con- 
trol between processors and to demand page the virtual address space between processors. 
Two approaches to reduce the overhead of the network communications are to employ faster, 
lower-overhead network protocols and to reduce the number of network operations. These 
two topics are discussed in the following sections. 


6.2.1. Network Protocols 

Our current CLASP prototype uses the TCP/IP network protocol to communicate 
between client and server processes. TCP/IP provides a full-duplex, error-free communica- 
tions channel between two endpoints. The protocol achieves these features at a cost in 
throughput and latency. However, the other available protocol (UDP/IP) does not provide 
reliable delivery of messages. 

CLASP does not require a stream connection. The CLASP network operations can be 
mapped directly onto a protocol that provides guaranteed delivery of messages. Such proto- 
cols might provide low latency messages, which would improve CLASP network paging 
times. 


6.2.2. Network Paging Performance 

Another approach to reducing the network overhead of a CLASP system is to reduce the 
traffic across the network. In this section, we describe two approaches to reducing the page 


traffic between CLASP clients and servers. 
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The UNIX system dynamically extends a program’s stack segment to accomodate the 
calling patterns of that program. The kernel never reduces the size of the stack segment, 
even if the pages are no longer used by the application program. In our prototype, this gen- 
erates unnecessary page traffic. If the server extends the stack onto a page that has been 
used previously by the client, the server retrieves the page through the network. This page, 
which is about to be overwritten with new data, could have been a fill-on-demand page and 
serviced locally. To eliminate this page traffic, the CLASP system could remove the pages 
beyond the stack pointer at each CAPC call and return. These pages, located beyond the 
current stack pointer, should be unused and can be discarded. 

Our CLASP prototype maintains a single copy of each page in the virtual address space. 
This simplifies the page management scheme but increases paging activity. Kai Li and Paul 
Hudak describe a virtual memory system for loosely-coupled systems [61]. Their system 
allows multiple instances of each page in the address space. Extra copies of a page are 
marked read-only. Attempts to modify these pages generate traps to the operating system 
that invalidate the extra copies of the page and proceed with the updates. This approach 
allows read-only pages (and pages that are read often and written seldom) to be replicated on 
the appropriate processors. 

6.3. Conclusions 

In this thesis, we introduce the Cross-Architecture Procedure Call or CAPC. The 
CLASP software architecture uses CAPCs to provide access to compute servers. The CAPC 
is a transparent mechanism to transfer a control thread between two processors. Unlike its 
predecessor, the Remote Procedure Call, CAPCs allow local and remote procedures to com- 
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municate through shared global variables including pointer data types. This allows existing 
programs, that use these constructs, to be partitioned between client and server processors 
using CAPCs without any source code changes. 


In the first chapter, we propose criteria for our new architecture. These criteria are: 


• The user need not restructure or recode his applications. 

• The programmer can specify an application’s partitioning. Changes to this par- 
titioning do not require changes to the application source code. 

• Interactive tasks execute on the workstation. That is, the workstation is not 
used as a simple terminal to submit jobs to the supercomputer. 

• CPU-intensive tasks execute on the supercomputer. 

• Optimization techniques, such as vector operation and parallel operations, 
specific to certain architectures are still useful for code segments executed on 
those architectures. 

• The compilers for each system need not be modified; a modified loader combines 
the output from the respective compilers into an executable file. 

• The operating system resolves issues of control transfer and data transfer 
between systems. 


CLASP meets these criteria. CLASP satisfies the first four criteria because it provides a 
transparent interface between routines on different processors. Routines on different proces- 
sors can pass pointer data types and share global variables. This transparency allows users 
to place routines on the architecture best suited for those routines. CLASP allows each 
architecture’s compilers to apply appropriate optimization techniques to routines that will 
execute on those processors. Our prototype does not modify existing compilers; it uses a new 
loader to combine object files for each architecture into a multi-architecture executable file. 
The CAPC runtime implementation is handled within the operating system. The operating 
system transparently handles paging traffic between local and remote processors. 
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Unlike the Remote Procedure Call, the Cross-Architecture Procedure Call does not res- 
trict the interface between procedures. CAPCs model the procedure call interface more com- 
pletely than RPCs. This feature allows existing applications to be re-compiled for a multi- 
archiecture environment and yield improved performance without any source code changes. 

The Cross-Architecture Procedure Call is an elegant mechanism for accelerating specific 
portions of applications programs. It extends a simple process model onto a new foundation 
that provides improved performance without introducing restrictions on calls between pro- 
cedures, access to global variables, and passing pointers. 
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APPENDIX A. 

ARGUMENTS AND RETURN VALUES 


This section describes mechanisms for passing arguments, determining the size of the 
argument list, and how return values are handled. This is a survey of several architectures 
and includes: how to determine the argument list location and size when acting as client, 
how to copy and use the argument list location and size when acting as a server, how to pack- 
age the return results when acting as a server, and how to interpret the return results when 
acting as a client. 

Each of the systems described in this appendix use the same data representation. They 
all use the IEEE floating point standard internally. They all store integer values with the 
same byte ordering. 

A.1. Motorola 68000 

The Motorola 68000 architecture does not load a register with the address of the argu- 
ment list as part of the procedure call instruction. Instead, the convention is to place the 
arguments on the stack. The subroutine call instruction pushes the return address on the 
stack. At procedure entry, the argument list is located 4 bytes above the current stack 
pointer. Figure A.l shows the 68000 stack frame at procedure entry. 


PRECEDING PAGE BLANK NOT FILMED 


— USL — INTENTIONALLY BLANK 
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argument n 


argument n-1 


... 


argument 2 


argument 1 


return address 




Stack Pointer 


Figure A.1 

Motorola 68000 Stack Frame 
(at procedure entry) 


Compilers for the 68000 architecture generate code that executes a link instruction as 
the first instruction of a subroutine. The LINK instruction pushes the contents of a specified 
register onto the stack and loads that register with the value of the stack pointer. The regis- 
ter modfied by the LINK instruction is then used as a base register, or frame pointer, to 
access both arguments and local variables for that procedure. The agument vector starts 8 
bytes above the value in this register (traditionally register A6). The first 4 bytes above A6 
are the previous contents of A6; the next 4 bytes are the return address. The stack after the 
procedure preamble is shown in figure A.2. 

The CLASP client routines for the 68000 architecture determine the length of the argu- 
ment vector by examining the instruction after the procedure call. Because the hardware 
does not provide a mechanism for including the length of the argument vector as part of the 
procedure call instruction, this instruction pops any arguments from the stack. The CLASP 
kernel decodes this instruction to determine the length of the argument vector. If the next 
instruction does not pop arguments from the stack, the CLASP kernel assumes the procedure 
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argument n 


argument n-1 


argument 2 


argument 1 


return address 


previous A6 value 


local variables 


A6 


Stack Pointer 


Figure A.2 

Motorola 68000 Stack Frame 
(after procedure prolog) 


has no arguments. 

The CLASP server routines for the 68000 architecture copy the argument vector and 
push the return address to provide the called procedure with a stack that appears to have 
been generated by the 68000 subroutine call instruction. 

Procedures and functions return their results in the DO and Dl registers. Most func- 
tions return their values in the 32-bit DO register. Floating point results are returned as 64 
bit values. For floating point results, both DO and Dl registers are used. 

A.2. Alliant FX Series 

The Alliant FX series is multiprocessor system that contains several computation ele- 
ments and I/O processors. The I/O processors are members of the Motorola 68000 family. 
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The computation elements provide a superset of the 68000 instruction set. Both processors 
use the same stack formats and calling sequences. The Motorola 68000 section of this appen- 
dix contains more information about this stack format, how to determine the location and 
size of the argument vector, and how return values are stored. 

A. 3. Convex C-l 

The Convex C-l uses a register, the argument pointer, to pass arguments to called sub- 
routines. The calling routine builds an argument vector and sets the argument pointer to 
point at the base of this vector. Called routines access arguments as offsets from this pointer. 
Figure A.3 shows the C-l stack frame at procedure entry, just after the procedure prologue 
has allocated storage for local variables. 

CLASP client routines determine the location of the argument vector from the contents 
of this register. C-l compilers store the argument length, as a count of 4 byte words, at the 
address just below the argument pointer. CLASP client routines determine the argument 
vector length from the value at this location. CLASP server routines load the argument 
register with the passed value. 

The C-l stores return values in the SO register. This 64 bit register contains all return 


values. 
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Callers RTN Address 


Caller LSI, part 2 


Caller Automatic Storage 

Arg N 

... 

Arg 1 

Arg Count (words) 


Callee LSI, part 1 

Saved S registers 

Saved A registers 

Saved PSW 

Return Address 

Callee LSI, part 2 


Callee Automatic Storage 


Caller FP 


AP 


Callee FP 
SP 


Figure A.3 

Convex C-l Stack Frame 
(after procedure prolog) 


A.4. IBM RT 

The IBM RT-PC presents arguments to subroutines as an array of bytes on the stack. 
For efficiency reasons, the first 4 arguments are passed in general registers t2 through r5 
respectively. For subroutines with only a few arguments, this convention improves perfor- 
mance; fewer memory operations are required to pass arguments to the subroutine. For rou- 
tines that take the address of any of these first four arguments, the called procedure’s prolo- 
gue saves them in a reserved area in the called procedure’s stack frame. A multi-word struc- 
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ture might be split; the first several words of the structure may be pased in one or more 
registers, the rest may be placed on the stack. After the called procedure moves these regis- 
ters to memory, they form a contiguous argument vector with the other arguments. On the 
IBM RT, the general register rJ is used as a stack pointer. The IBM RT stack at procedure 
entry is shown in figure A.4. 

Upon entry, the called procedure adjusts the stack pointer to reserve space for local 
variables and temporary space. The IBM RT does not use the stack in a true stack-oriented 
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Figure A. 4 

IBM RT-PC Stack Frame 
(at procedure entry) 
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fashion. Instead of pushing and popping values as needed, the stack is extended to the max- 
imum depth required by the procedure and left there. Local variables are referenced relative 
to the R13 register; parameters to other subroutines are referenced relative to the Rl regis- 
ter. Figure A.5 shows the stack frame after the procedure prologue has executed. 

The IBM RT returns values in registers R2 and R3. Simple values, those of 32 or fewer 
bits, are completely contained in R2. Floating point values, which are passed as 64 bit quan- 
tities, are passed in both R2 and R3. 
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Unfortunately for our goals, the compilers on the IBM RT use different conventions for 
returning structures and passing procedures as formal arguments than we have seen. These 
differences make a CLASP system between our SUNs and the RT impossible. 
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APPENDIX B. 

PERFORMANCE MEASUREMENTS 


This appendix describes the performance of several test programs using our CLASP pro- 
totype kernel. We compare the execution times using our CLASP kernel against the execu- 
tion time for the program with a standard UNIX kernel. 

B.l. Double Precision UNPACK Benchmark 

We obtained a copy of the LINPACK linear systems library from Argonne National 
Laboratories [36]. This package contains FORTRAN subroutines to solve the equation: 

Ax = b 

A benchmark program included with the library performs a number of iterations generating, 
factoring, and solving a matrix. The A and b matrices are dimensioned to 200 elements. 
Another variable determines the size of the system to be solved. We partitioned the FOR- 
TRAN program into separate modules. The LINPACK routines to factor and solve the sys- 
tem execute on the server processor. The matrix generation routine and benchmark harness 
execute on the client processor. 

We ran the benchmark for systems whose order ranged from 5 through 200. For each 
size, we collected the following statistics: client user and system time, server user and system 
time, total time, page traffic, number of calls, and the time for a non-CLASP version. Each 
execution made 53 CAPC calls and returns. The number of calls is a function of the bench- 


mark itself, not the size of the system. 



The graph in figure B.l shows the execution times for the UNPACK benchmark. Fig- 
ure B.3 shows the number of pages moved between client and server for each benchmark run. 

Figure B.2 shows the speed ratio between client and server to recover the CLASP over- 
head. Matrices of smaller order than 59 incur more overhead than can be made up on any 
server. At 59, the overhead can be offset with a processor that is approximately 15 times fas- 
ter than the client. When the matrix is of order 81, the server need be only twice as fast as 


the client to recover the overhead. 
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Figure B.2 

Speedup required to pay for CAPC overhead 
Large dimensioned Arrays 


In figure B.3, the paging traffic appears to grow linearly with the size of the system. 
The overdimensioned matrices interact with the page size to produce this behavior. Each 
page holds more than one row of the matrix. A page fault moves at least an entire 200 ele- 
ment row, even though only the first 5 columns of that row will be used. Each page holds 
more than one row of the matrix. For matrices of order 40 through 80, we can discern the 
steps in the page traffic. 

This benchmark uses over-dimensioned arrays. Therefore, array accesses are spread 
over a larger section of the address space. In the CLASP environment, the sparse use of the 
address space results in extra paging overhead. Figure B.4 shows the breakeven points for a 
version of this benchmark that uses arrays dimensioned to the exact size of the problem being 
solved. Because the smaller array is stored in a more compact section of the address space, 



page 

traffic 



Order of System Solved 


Figure B.3 

UNPACK Benchmark Network Paging 
Arrays Declared for 200 Elements 


the paging traffic is lower than in the original benchmarks. This reduces the breakeven 
point. With this modification, the CLASP system can break even as soon as the system is 
order 42 — instead of order 59. 

We believe that this benchmark does not show the true advantages of our demand- 
paging system. The benchmark builds and factors the A matrix a number of times. The 
server does the factoring; the client rebuilds the matrix. This causes additional paging over- 
head. We feel that a more realistic situation is where the A matrix is factored once and then 
used to solve the system for many different values of b. 


Figure B.5 shows the pages transferred during virtual time intervals for the execution of 
the benchmark for a 75x75 matrix. Each timeslot represents 50 milliseconds of processor 
time. The paging traffic is concentrated in short bursts. Half of the paging traffic occurs in 
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I 
I 

| approximately 7 % of the virtual time for this program. 

| Figure B.6 shows the number of times each page moved between processors. This graph 

j does not include the movement of the stack page; the only stack movement was for a single 

page to the server. Each point on the graph represents a page of 8192 bytes. 
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B.2. Compression Program 

We partitioned the compress program from the 4.3 BSD distribution into client and 
server routines [87]. This program uses a modified Lempel-Ziv algorithm to generate codes 
for common substrings and replace them in the compressed file. 

A profiling run showed that the program spends much of its time in two routines: 
compress () and outputQ. We built a version of the program with these two routines on the 
server architecture. 

We used the partitioned program to compress a copy of our UNIX kernel. The program 
compressed this 472,689 byte file into 296,314 bytes. We used the UNIX gproj{ 1) utility to 
determine what portions of the program used the most CPU time. The compress( ) function 
was invoked once and used 11.7 seconds of CPU time. The outputQ function was invoked 
159,280 times and used another 7.13 seconds of CPU time. Although each invocation of the 
outputQ subroutine was too short to make a CAPC advantageous, all but two of these invoca- 
tions came from the compressQ routine — which is on the same processor as the output rou- 
tine. OutputQ made a small number of calls to routines on the client processor (39 calls in 
this instance). 

We built a version of the compress program with the compressQ and outputQ functions 
on the server processor. For the data files we ran, the partitioned program’s overhead was 
larger than the execution time of the original program. 

Figure B.7 shows which pages were moved and how often. Only two stack pages moved 
— each moved 1 time. The stack frames are not included in this graph. Page 16 moves 
because most of the static variables are on that page. The client and server routines are 
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accessing different variables that happen to be on the same page. A more sophisticated 
loader might place these variables on different pages to reduce this contention. 

Figure B.8 shows the actual paging behavior of the program. This depicts the pages 
transferred during each time slot. Each timeslot is 50 milliseconds of user time; the time 
spent transferring pages between hosts is not included. Like the partitioned LINPACK 
benchmark, this program generated most of its network page faults in a short time period. 
Half of the page faults were generated in approximately 5% of the virtual time. 
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APPENDIX C. 
CODE SAMPLES 


This appendix contains sample code segments for routines that demonstrate some 
features of the CLASP architecture. 

The first sample shows how local and remote procedures can be passed as formal 
paramters to other procedures. There are no special actions to differentiate between local 
and remote procedures. 

The second example shows a program that builds and traverses a tree structure. The 
program passes pointers between the client and server; as the program traverses the tree, the 
demand paging system moves parts of the tree as the program accesses them. The routines 
have the same structure and arguments as they would if compiled for a more traditional 
single-processor system. 

C.l. Procedures aa Formal Parameters 

This example demonstrates how the CLASP system allows the applications to pass pro- 
cedures as formal parameters. No special compilation techniques are required to account for 
client and server differences. Both local and remote procedures are stored in the argument 
list using the same representation. The called procedure does not require special operations 
to differentiate between local and remote formal procedures. Figures C.l and C.2 show the 
client and server portions of a program that passes procedures as formal parameters. 


extern int f oosquare () , f oocube () , f oo Q ; 




main () 

{ 

lnt 1,J; 

for (1=0; 1 < 10; 1++) 

{ 

3 = f oo (f oosquare , 1) ; 
3 = f oo (f oocube , 1) ; 

> 

exit (0) ; 


lnt f oosquare (1) lnt 1; 

{ 

return (1*1); 

> 

Figure C.l 

Formal Procedures — Client Side 



I 

I 


lnt foo (proc, arg) 

lnt (*proc) () ; /* procedure parm */ 

lnt arg ; 

< 

return ((*proc) (arg)); 

> 


I 

I 


f oocube (arg) lnt arg; 

< 

return (arg * arg * arg) ; 

> 

Figure C.2 

Formal Procedures — Server Side 
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C.2. Pointer Structures 

This sample program builds and traverses binary trees. Some of the tree manipulation 
routines execute on the client; others execute on the server processor. The code in this exam- 
ple was written for a single processor system. 

There were no changes to the source code to make it run in a CAPC environment. We 
only changed the linking phase of the compilation process to use our new loader. Figure C.3 
contains the main section of the program, which executes on the client. The code in figure 
C.4 performs several operations on the tree. This code also executes on the client. The code 
in figure C.5 traverses the tree in postorder and preorder. These two routines execute on the 


server processor. 


main (argc , argv) 
int argc ; 

char **argv; 

{ 

int i, value, parms; 

char buf[128], cmd; 

static struct node root; 

while (printf ("CMD: ") , fflush (stdout) , (gets (buf) != NULL)) 

< 

if (strlen (buf) = 0) break; 

parms = sscanf (buf, "%c %d", ftcmd, ftvalue) ; 

switch (cmd) 

< 

case *1’: 

inorder (&root) ; break; 
case ’L’: 

preorder (fcroot) ; break; 
case ’R’: 

postorder (ftroot) ; break; 
case ’A': 

i = insert (value, fcroot) ; 

printf ("value %d, now has %d hlts\n", value, i) ; 
break; 
case ’F*: 

i = find (value, fcroot) ; 

printf ("value %d has %d hits\n", value, i) ; 
break; 
case ’Q’: 

goto quit; 

> 

> 

quit: 

exit (0) ; 


> 


Figure C.3 

Pointers in a CAPC environment — Main code 



find (value, root.) int value; struct node *root; 

If (root = (struct node *) NULL) return (-1) ; 
if (root -> value = value) return root -> hits; 

If (root -> value > value) return find (value, root -> left); 

If (root -> value < value) return find (value , root -> right) ; 
return (-1) ; 

> 

Insert (value, root) Int value; struct node *root; 

< 

If (root = (struct node *) NULL) exit (1) ; 
if (value = root -> value) return (++(root -> hits)); 
if (value < root -> value) { /* down left side */ 

if (root -> left = (struct node *) NULL) { 

root -> left = (struct node *)malloc (sizeof (struct node)); 
root -> left -> value = value; 
return (root -> left -> hits = 1) ; 

> 

return Insert (value, root -> left); /* recurse */ 

y 

If (value > root -> value) { /* down right side */ 

if (root -> right = (struct node *) NULL) { 

root -> right = (struct node *)malloc (sizeof (struct node)) ; 
root -> right -> value = value; 
return (root -> right -> hits = 1) ; 

> 

return Insert (value, root -> right); /* recurse */ 

> 

return (-1) ; 

> 

inorder (root) struct node *root; 

{ 

if (root = (struct node *) NULL) return; 
inorder (root -> left) ; 

printf ("%d: 9?d hits\n" , root -> value, root -> hits); 
inorder (root -> right) ; 

> 

Figure C.4 

Pointers in a CAPC environment — Client Code 
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#include <stdio.h> 

#include "node . h" 

postorder (root) 
struct node *root; 

{ 

If (root = (struct node *) NULL) 
return ; 

postorder (root -> left) ; 
postorder (root -> right) ; 

prlntf ("SSd: %& hits\n", root -> value, root -> hits); 

> 

preorder (root) 
struct node *root; 

< 

If (root = (struct node *) NULL) 
return; 

prlntf ("95d: %d hlts\n", root -> value, root -> hits); 
preorder (root -> left) ; 
preorder (root -> right) ; 

> 

Figure C.5 

Pointers in a CAPC environment — Server Code 
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APPENDIX D. 

CLASP CONFIGURATION AND LOG FILES 


Our prototype uses a static configuration table to assign server processors. The file 
/usr/local/etc/claspd.config contains the configuration data. The file contains lines that 
describe how much logging information to generate and addresses for servers of appropriate 
architectures. Figure D.l shows a sample configuration file. 

Our configurations usually store logging information in the files /usr/adm/ claspd.log 
and /u sr/adm/ clasp d.pr of. The claspd.log file contains high level information describing the 
current host addresses for specific architectures and the starting and finishing times for server 
processes. Figure D.2 shows a segment from this file. 

Claspd.prof provides more detailed information. This file records page traffic and call 
behavior. At normal logging levels, the kernel stores summary data in this file. For each 
client and server on the local host, the file contains the number of CAPC calls and returns 
and the number of pages moved across the network. More detailed logging generates a line 
for each CAPC call, CAPC return or page transfer. All lines are marked with the current 
time and process identifier. A segment of this file is shown in figure D.3 



# configuration file for CLASP kernel. 

# The daemon reads this file at startup and whenver it receives 

# a SIGHUP signal. 


# Syntax: 

# 


profiling 


level 


hostname 


missing hostname defaults to localhost 
missing level defaults to 2 

logging {on I off} pathname 

missing pathname leaves it unchanged, 
must specify on/off field. 


profiling 

logging 


brutus . cs . uiuc . edu 
/usr/adm/claspd . log 


server architecture hostname 
architecture is integer 
hostname is string 
gotta specify both. 


# 10 = M_RBE1 
server 10 


crl.cs.uiuc.edu 


# 20 = M_68020R 

server 20 brutus.cs.uiuc.edu 


Figure D.l 

Sample Claspd Configuration File 
/ usr /local /etc / claspd. config 
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Fri Mar 27 11:04:15 1987: daemon 88: Re-initialize server tables 

Frl Mar 27 11:04:17 1987: daemon 88: 0x10 @ brutus.cs.uluc.edu 

Fri Mar 27 11:04:19 1987: daemon 88: 0x20 @ brutus.cs.uluc.edu 

Frl Mar 27 20:55:05 1987: server 1274: client at 192.17.238.2/1053 

Fri Mar 27 20:55:19 1987: server 1274: exit/sig 0/11. 

user/sys 0.100000/6.960000 secs 

Sun Mar 29 11:29:39 1987: server 3148: client at 192.17.238.2/1137 

Sun Mar 29 11:30:11 1987: server 3148: exit/sig 0/11. 

user/sys 0.000000/0.960000 secs 

Figure D.2 
Sample claspd.log 


I 

I 

I 

I 


0x73f8 


037398/400001 pid 3147: call to server at 
037398/400003 pld 3147: pageout at OxeffcOOO 
037398/460002 pld 3147: pageout at 0x20000 
037398/520001 pld 3147: pageout at 0x28000 
037398/580000 pld 3147: pageout at 0x2a000 
037398/640000 pld 3147: call from server to 0x326c 
037398/640001 pld 3147: pagein at 0x20000 
037398/700000 pld 3147: pagein at 0x28000 
037398/740001 pld 3147: return to server at 0x7448 
037398/760002 pld 3147: pageout at 0x28000 
037398/820000 pld 3147: pageout at 0x2c000 
037398/860001 pld 3147: call from server to 0x326c 
037398/860002 pld 3147: pagein at 0x28000 
037398/920001 pld 3147: return to server at 0x7448 
037398/940000 pld 3147: pageout at 0x28000 
037398/980001 pld 3147: call from server to 0x326c 
037399/000000 pld 3147: pagein at 0x28000 
037399/040001 pld 3147: return to server at 0x7448 
037399/060001 pid 3147: pageout at 0x20000 


037409/420000 pld 3147: return from server to 0x21f8 

037411/020000 pld 3147: rl : CAPC calls/returns: local 0/0 network 2/9 
037411/020001 pld 3147: rl: CAPC pagelns 11, pageouts 15 


Figure D.3 
Sample claspd.log 
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